Richard Seroter's Architecture Musings

Author: Richard Seroter

  • Interview Series: Four Questions With … Brian Loesgen

    Happy December and welcome to my 15th interview with a leader in the “connected technology” space.  This month, we sit down with Brian Loesgen who is a prolific author, blogger, speaker, salsa dancer, former BizTalk MVP, and currently an SOA architect with Microsoft.

    Q: PDC 2009 has recently finished up and we saw the formal announcements around Azure, AppFabric and the BizTalk Server roadmap.  It seems we’ve been talking to death about BizTalk vs. Dublin (aka AppFabric) and such, so instead, talk to us about NEW scenarios that you see these technologies enabling for customers.  What can Azure and/or AppFabric add to BizTalk Server and allow architects to solve problems easier than before?

    A: First off, let me clarify that Windows Server AppFabric is not just “Dublin”-renamed, it brings together the technologies that were being developed as code-name “Dublin” and code-name “Velocity”. For the benefit of your readers that may not know much about “Velocity”, it was a distributed in-memory cache, ultra-high performance and highly-scalable. Although I have not been involved with “Velocity”, I have been quite close to “Dublin” since the beginning.

    I think the immediate value people will see in Windows Server AppFabric is that now.NET developers are being provided with a host for their WF workflows. Previously developers could use SharePoint as a host, or write their own host (typically a Windows service). However, writing a host is a non-trivial task once you start to think about scale-out, failover, tracking, etc. I believe that the lack of a host was a bit of an adoption blocker for WF and we’re going to see that a lot of people that never really thought about writing workflows will start doing so. People will realize that a lot of what they write actually is a workflow, and that we’ll see migration once they see how easy it is to create a workflow and host it in AppFabric and expose it as WCF Web services. This doesn’t, of course, preclude the need for integration server (BizTalk) which as you’ve said we’ve been talked to death about, and “when to use what” is one of the most common questions I get. There is a time and place for each, they are complementary.

    Your question is very astute, although Azure and AppFabric will allow us to create the same applications architected in a new way (in the cloud, or hybrid on-premise-plus-cloud), they will also allow us to create NEW types of applications that previously would not have been possible. In fact, I have already had many real-world discussions with customers around some novel application architectures.

    For example, in one case, we’re working with geo-distributed (federated) ESBs, and potentially tens of thousands of data collection points scattered around the globe, each rolling up data to its “regional ESB”. Some of those connection points will be in VERY remote locations, where reliable connectivity can be a problem. It would never have been reasonable to assume that those locations would be able to establish secure connections to a data center with any kind of tolerable latency, however, the expectation is that somehow they’ll all be able to reach to cloud. As such, we can use the Windows Azure platform Service Bus as a data collection and relay mechanism.

    Another cool pattern is using the Windows Azure platform  Service Bus as an entry point into an on-premises ESB.  In the past, if a BizTalk developer wanted to accept input from the outside typically they would expose a Web service, and reverse-proxy that to make it available, probably with a load balancer thrown in if there was any kind of sustained or spike volume. That all works, but it’s a lot of moving parts that need to be set up. A new pattern we can do now is that we can use the Windows Azure platform Service Bus as a relay: externals parties send messages to it (assuming they are authorized to do so by the Windows Azure platform Access Control service) , and a BizTalk receive location picks up from it. That receive location could even be an ESB on-ramp. I have a simple application that integrates BizTalk, ESB, BAM, SharePoint, InfoPath, SS/AS, SS/RS (and a few more things). It was trivial for me to add another receive location that picked up from the Service Bus (blog post about this coming soon, really J).  Taking advantage of Azure to act as an intermediary like this is a very powerful capability, and one I think will be very widely used.

    Q: You were instrumental in incubating and delivering the first release of the ESB Toolkit (then ESB Guidance) for Microsoft.   I’ve spoken a bit about itineraries here, but give me your best sales pitch on why itineraries matter to Microsoft developers who may not familiar with this classic ESB pattern.  When is the right time to use them, why use an itinerary instead of an orchestration, when should I NOT use them?

    A: I recently had the pleasure of creating and delivering a “Building the Modern ESB” presentation with a gentleman from Sun at the SOA Symposium in Rotterdam. It was quite enlightening, to see the convergence and how similar the approaches are. In that world, an itinerary is called a “micro-flow”, and it exists for exactly the same reasons we have it in the ESB Toolkit.

    Itineraries can be thought of as a light-weight service composition model. If all you’re doing is receiving a request, calling 3 services, perhaps with some transformations along the way, and then returning a response, then that would be appropriate for an itinerary. However, if you  have a more complex flow, perhaps where you require sophisticated branching logic or compensation, or if it’s long running, then this is where orchestrations come into play.  Orchestration provides the rich semantics and constructs you need to handle these more sophisticated workflows.

    The ESB Toolkit 2.0 added the new Broker capabilities, which allow you to do conditional branching from inside an itinerary, however I generally caution against that as its one more place you could hide business logic (although there will of course be times where this is the right approach to take).

    A pattern that I REALLY like is to use BizTalk’s Business Rules Engine (BRE) to dynamically select an itinerary and apply it to a message. The BRE has always been a great way to abstract business rules (which could change frequently) from the process (which doesn’t change often). By letting the BRE choose an itinerary, you have yet another way to leverage this abstraction, and yet another way you can quickly respond to changing business requirements.

    Q: You’ve had an interesting IT career with a number of strategic turns along the way.  We’ve seen the employment market for developers change over the past few years to the point that I’ve had colleagues say that they wouldn’t recommend that their kids go into computer science and rather focus on something else.  I still think that this is a fun field with compelling opportunities, but that we all have to be more proactive about our careers and more tuned in to the skills we bring to the table. What advice would you give to those starting out their IT careers with regards to where to focus their learning and the types of roles they should be looking for?

    A: It’s funny, when the Internet bubble burst, a couple of developers I knew then decided to abandon the industry and try their hands at something else: being a mortgage broker. I’m not sure where they are now, probably looking for the next bubble J

    You’re right though, I’ve been fortunate in that I’ve had the opportunity to play many roles in this industry, and I have never been as excited about where we are as an industry as I am now. The maturing and broad adoption of Web service standards, the adoption of Service-Oriented Architectures and ESBs, the rapid onset of the cloud (and new types of applications cloud computing enables)… these are all major change agents that are shaking up our world. There is, and will continue to be, a strong market for developers. However, increasingly, it’s not so much what you know, but how quickly you can learn and apply something new that really matters.  In order to succeed, you need to be able to learn and adapt quickly, because our industry seems to be in a constant state of increasingly rapid change. In my opinion, good developers should also aspire to “move up the stack” if you want to advance your career, either along a technical track (as an architect) or perhaps a more business-related track (strategy advisor, project management). As you can see from the recently published SOA Manifesto (http://soa-manifesto.org), providing business value and better alignment between IT and business are key values that should be on the minds of developers and architects, and SOA “done right” facilitates that alignment.

    Q [stupid question]: year you finally bit the bullet and joined Microsoft.  This is an act often equated with “joining the dark side.”  Now that phrase isn’t only equated with doing something purely evil, but rather, giving into something that is overwhelming and seems inevitable such as joining Facebook, buying an iPhone, or killing an obnoxious neighbor and stuffing him into a recycling container.  Standard stuff.  Give us an example (in life or technology) of something you’ve been resisting in your life, and why you’re holding out.

    A: Wow, interesting question. 

    I would have to say Twitter. I resisted for a long time. I mean c’mon, I was already on Facebook, Live, LinkedIn, Plaxo…. Where does it end? At some point you need to be able to live your life and do your work rather than talking about it. Plus, maybe I’m verbose, but when I have something to say it’s usually more than 140 characters, so I really didn’t see the point. However, I succumbed to the peer pressure, and now, yes, I am on Twitter.

    Do you think if I say something like “you can follow me at http://twitter.com/BrianLoesgen” that maybe I’ll get the “Seroter Bump”? 🙂

    Thanks Brian for some good insight into new technologies.

    Share

  • Using UML vs. ADL (Arbitrary Diagramming Language)

    At my company, we highly encourage our architects and developers to communicate their design models through UML.  We have a standard set of diagrams that we regularly use to convey aspects of a given solution, including:

    • Component diagrams for system dependency, functional decomposition and data flow
    • Use case diagrams for business and system scenarios
    • Sequence diagrams for a timing perspective on system interactions
    • Deployment diagrams for both logical and physical representations of the hardware/software landscape

    We all know that the job of architect pretty much consists of “ask lots of annoying questions” and “draw boxes and lines.”  The downside with being so UML-centric is that UML can accidentally become the default way to draw boxes and lines that represent all sorts of things.  My boss has encouraged us to be observant for when a particular graphical representation should take a more “marketecture” approach (i.e. fancy boxes and lines in PowerPoint or Visio) vs. formal modeling in UML or BPMN.  She labeled the PowerPoint/Visio approach as using the Arbitrary Diagramming Language (ADL) method.  Seems about right.

    What’s an example of this?  I’ll give you two.  I recently had to put together a reference architecture for a project I’m working on.  My first inclination was to throw together a functional decomposition diagram (component diagram) of all the relevant functional pieces.  However, this was very early in the project lifecycle, and I knew that this diagram would be consistently referred to in both business and technical meetings.  So instead of going nuts on a UML diagram that would be utterly confusing, and potentially hacked up to represent certain ideas, I went with a PowerPoint model that has been in regular use for 5 months now.

    In another case, I’m trying to represent deployment options (incremental phased approach vs. all-at-once) for a global system.  I began this effort thinking about component diagrams, deployment diagrams, swim lanes and mashing some things together.  Again though, I’d probably have to bastardize UML and overload agreed-upon constructs in order to convey my point.  So, I went again to trusty PowerPoint and modeled out the options in a more business-friendly way.  I could create my own terminology/model without feeling guilty for screwing up standard UML.

    This way, I could convey a sequence of deployment phases (through a set of slides), which parties were involved, and the necessary interim interfaces without butchering UML.

    So when to use each modeling style?  I’d like your thoughts as well, but for me, I switch to an ADL / marketecture model when:

    • Doing early project stage diagrams to convey concepts presented to broad audiences (e.g. logical architectures, deployment diagrams, proposed data flow between systems/organizations)
    • I realize that I need to cannibalize, bastardize, overload or otherwise distort UML notation to convey a point (e.g. user experience flow, etc)

    I fire up a UML tool for modeling when:

    • I’m building lasting representations of how a system will be built and providing an accurate blueprint to the development team
    • Conveying solution aspects to solely technical teams through presentations or design documents
    • I can take a particular solution aspect and clearly convey the point via one or many complimentary UML model perspectives.

    The hard part is not stuffing so much into a single visual model as to render it useless.  There’s no shame in requiring two (or more) different perspectives on a problem.

    So there you go. Is Visio/PowerPoint/other-boxes-and-lines-tool-of-choice your first place to go when you have to diagram something for your project team or management?  Or do you try and shape and morph traditional diagramming notations to fit your needs?

  • Validating Incoming Data Using the BizTalk Business Rules Engine

    A project team recently asked me if they could use the BizTalk BRE to validate incoming data.  I asked what sort of validation we were talking about, and it came down to four areas:

    • Setting default values when the incoming fields were empty
    • Doing look-ups based on existing data and populating related fields
    • Doing concatenation of fields
    • Catching any business validation errors and recording them

    Before answering “of course it can”, I figured I’d quickly roll up my sleeves and prove that the BRE can do those things fairly easily.

    The Setup

    First, I had to get my environment set up.  This started with a BizTalk XSD schema to represent the data coming into the BRE.  In this scenario, the content is actually metadata about a physical file that arrives into the company.

    Notice that I have a “validation errors” record at the bottom with a repeating “error” element.

    Next, I have a really simple database table that can be used for lookups.  Based on providing a “study” and “product” value (which comes in via the inbound XML), I want to be able to figure out which “investigator” to add to the message.

    Next I need a .NET class to perform a few functions that the BRE doesn’t provide for me out of the box.  First, I need to be able to concatenate two strings.  Yes, it’s ridiculous that I have to write an operation to do this vs. the BRE having something built in.  Deal with it.

    public class RuleFunctions
{
   public string ConcatValues(string input1, string input2)
   {
      return input1 + input2;
   }
}

    Next, I need a function that can add nodes to my XML message when validation errors are discovered.  After adding references to the Microsoft.RulesEngine.dll and System.Xml, I wrote the following quick-and-dirty code.

    public void AddValidationError(TypedXmlDocument activeDoc, string err)
{
   XmlDocument doc = activeDoc.Document.OwnerDocument;

   XmlNode errorRoot = doc.SelectSingleNode("//ValidationErrors");

   XmlElement newError = doc.CreateElement("Error");
   newError.InnerText = err;

   errorRoot.AppendChild(newError);
}

    I accept a “TypedXmlDocument”, yank out the underlying XmlDocument, grab a pointer to the “ValidationErrors” record, and append a new node containing whatever error message was defined within the BRE.

    After GAC-ing that helper DLL, I’m ready to roll.  Before building my rules in the BRE, I wanted to establish a friendly vocabulary for my XML nodes, database columns and helper functions.  So, I defined a new vocabulary and created definitions for all the “get”, “set” and “function” operations that my rules required.

    Setting Default Values

    The first requirement was to be able to set default values on nodes.  This one’s pretty easy.  I just check for either an empty string or null value, and set the default value.

    Doing Lookups

    The next requirement was to take values from the inbound message, perform a lookup and fill in an empty node from the message.  In the rule below, I say that if the value in the XML message equals a value from the database, and another value also equals a value from the database, then set the XML node (Investigator Name) equal to another column in the database.

    What this generates behind the scenes when you execute the rule is this:

    So the proper T-SQL is built and executed to grab my lookup value.

    Performing Concatenation

    There’s no out-of-box string concatenation function in the BRE, so here I use the custom class I built earlier.  I want this rule to fire at all times, so my condition is set to something that will always be true (unless math itself starts failing).  I take a static file path location and then append the file name of the document coming into my company and turn that whole thing into an absolute path to the file.

    Catching and Recording Business Exceptions

    The final requirement was to detect business validation errors and record them.  So, we can use my custom built function from earlier to create rules that look for business errors and add nodes to the message itself which outline the error details.

    Remember that my custom function takes in a “TypedXmlDocument” type, so I pass the document itself into the function by dragging the root node of my schema from the XML Schemas Fact Explorer view into the operation parameter.

    The Result

    So what happens when I pass in a document that flexes all these conditions? I start off with an instance document that only has a few values.


    So I should see some default values get set, the “investigator” node set based on database lookups, the “file path” node be populated with the concatenated value, and I should see some errors at the end because nodes like “document sub type” and “site” are empty.

    To test this rule set, I’m using the built in rule tester, so I have to pass in a valid XML instance, a database connection, and a fact creator that provides the BRE with an instance of my custom .NET class.

    When the rule policy is finished executing, my input XML is changed to this:

    You can see that all the rules we built earlier got applied successfully.  So yes, you can use the BRE to do some pretty easy-to-maintain business validation.  This may not be a fit if you’re dealing with a nightly batch of 25,000 records, but if you are doing a messaging based solution, the BRE can do some nice work for you.

    Share

  • My Presentation from Sweden on BizTalk/SOA/Cloud is on Channel9

    So my buddy Mikael informs me (actually all of us), that my presentation on “BizTalk, SOA and Leveraging the Cloud” from my visit to the Sweden User Group is finally available for viewing on Microsoft’s Channel9.

    In Part 1 of the presentation, I lay the groundwork for doing SOA with BizTalk, and, try to warm up the crowd with stupid American humor.  Then, in Part II I explain how to leverage the Google, Salesforce.com, Azure and Amazon clouds in a BizTalk solution.  Also, either out of sympathy or because my material was improving, you may hear a few more audible chuckles.  I take it where I can get it.

    I had lots of fun over there, and will start openly petitioning for a return visit in 6 months or so.  Consider yourself warned, Mikael.

    Share

  • Interview Series: Four Questions With … Lars Wilhelmsen

    Welcome to the 14th edition of my interview series with thought leaders in the “connected technology” space.  This month, we are chatting with Lars Wilhelmsen, development lead for his employer KrediNor (Norway), blogger, and Connected Systems MVP.  In case you don’t know, Connected Systems is the younger, sexier sister of the BizTalk MVP, but we still like those cats.  Let’s see how Lars holds up to my questions below.

    Q: You recently started a new job where you have the opportunity to use a host of the “Connected System” technologies within your architecture.  When looking across the Microsoft application platform stack, how do you begin to align which capabilities belong in which bucket, and lay out a logical architecture that will make sense for you in the long term?

    A: I’m Development Lead. Not a Lead Developer, Solution Architect or Develop  Manager, but a mix of all those three, plus that I put on a variety of other “hats” during a normal day at work. I work close with both the Enterprise Architect and the development team. The dev team consists of “normal” developers, a project manager, a functional architect, an information architect, an tester, a designer and a “man-in-the-middle” whose only task is to “break down” the design into XAML.

    We’re on a multi-year mission to turn the business around to meet new legislative challenges & new markets. The current IT system is largely centered around a mainframe-based system, that (at least as we like to think today, in 2009) has too many responsibilities. We seek to use “Components of the Shelf” where we can, but we’ve identified a good set of subsystems that needs to be built from scratch. The strategy defined by the top-level management states that we should seek to use primarily Microsoft technology to implement our new IT platform, but we’re definitely trying to be pragmatic about it. Right now, a lot of the ALT.NET projects gains a lot of usage and support, so even though Microsoft brushes up bits like Entity Framework and Workflow Foundation, we haven’t ruled out the possibility to use non-Microsoft components where we need to. A concrete example is in a new Silverlight –based application we’re developing right now; we evaluated some third party control suites, and in the end we landed on RadControls from Telerik.

    Back to the question, I think over time, we will see a lot of the current offerings from Microsoft, either it targets developers, IT Pro’s or the rest of the company in general (Accounting, CRM etc. systems) implemented in our organization, if we find the ROI acceptable. Some of the technologies used by the current development projects include; Silverlight 3, WCF, SQL Server 2008 (DB, SSIS, SSAS) and BizTalk. As we move forward, we will definitely be looking into the next-generation Windows Application Server / IIS 7.5 / “Dublin”, as well as WCF/WF 4.0 (one of the tasks we’ve defined in the near future is a light-weight service bus), and codename “Velocity”.

    So,the capabilities we’ve applied so far (and planned) in our enterprise architecture is a mix of both thoroughly tested and bleeding edge technology.

    Q: WCF offers a wide range of transport bindings that developers can leverage.  What are you criteria for choosing an appropriate binding, and which ones do you think are the most over-used and under-used?

    A: Well, I normally follow a simple set of “Rules of thumbs”;

    • Inter-process: NetNamedPipeBinding
    • Homogenous intranet communication: NetTcpBinding
    • Heterogeneous intranet communication: WSHttpBinding BasicHttpBinding
    • Extranet/Internet communication: WSHttpBinding or BasicHttpBinding

    Now, one of the nice thing with WCF is that is possible to expose the same service with multiple endpoints, enabling multi-binding support that is often needed to get all types of consumers to work. But, not all types of binding are orthogonal; the design is often leaky (and the service contract often need to reflect some design issues), like when you need to design a queued service that you’d eventually want to expose with an NetMsmqBinding-enabled endpoint.

    Often it boils down to how much effort you’re willing to put in the initial design, and as we all (hopefully) know by now; architectures evolves and new requirements emerge daily.

    My first advice to teams that tries to adapt WCF as a technology and service-orientation, is to follow KISS – Keep It Simple, Stupid. There’s often room to improve things later, but if you do it the other way around, you’ll end up with unfinished projects that will be closed down by management.

    When it comes to what bindings that are most over- and under-used, it depends. I’ve seen someone that has exposed everything with BasicHttpBinding and no security, in places where they clearly should have at least turned on some kind of encryption and signing.

    I’ve also seen highly optimized custom bindings based on WSHttpBinding, with every small little knob adjusted. These services tends to be very hard to consume from other platforms and technologies.

    But, the root cause of many problems related to WCF services is not bindings; it is poorly designed services (e.g. service, message, data and fault contracts). Ideally, people should probably do contract-first (WSDL/XSD), but being pragmatic I tend to advice people to design their WCF contracts right (if in fact, they’re using WCF). One of the worst thing I see, is service operations that accepts more than one input parameter. People should follow the “At most one message in – at most one message out” pattern. From a versioning perspective, multiple input arguments are the #1 show stopper. If people use message & data contracts correctly and implements the IExtensibleDataObject, it is much easier in the future to actually version the services.

    Q: It looks like you’ll be coming to Los Angeles for the Microsoft Professional Developers Conference this year.  Which topics are you most keen to hear about and what information do you hope to return to Norway with?

    A: It shouldn’t come as a surprise, but as a Connected Systems MVP, I’m most excited about the technologies from that department (Well, they’ve merged now with the Data Platform people, but I still refer to that part of MSFT as Connected Systems Div.). WCF/WF 4.0 will definitely get a large part of my attention, as well as Codename “Dublin” and Codename “Oslo”. I will also try to watch the ADFSv2 [Formerly known as Codename “Geneva”] sessions. Apart from that, I hope to use a lot of time talking to other people. MSFTies, MVPs and other people. To “fill up” the schedule, I will probably try to attend some of the (for me) more exoteric sessions about Axum, Rx framework, parallelization etc.

    Workflow 3.0/3.5 was (in my book) a more or less a complete failure, and I’m excited that it seems like Microsoft has taken the hint from the market again. Hopefully WF 4.0, or WF 3.0 as it really should be called (Microsoft product seems to reach maturity first at version 3.0), will hopefully be a useful technology that we’ll be able to utilize in some of our projects. Some processes are state machines, some places do we need to call out in parallel to multiple services – and be able to compensate if something goes wrong, and other places do we need a rule engine.

    Another thing we’d like to investigate more thorough, is the possibility to implement claims-based security in many of our services, so (for example) can federate with our large partners. This will enable “self service” of their own users that access our Line of Business applications via the Internet.

    A more long term goal (of mine, so far) is definitely to use the different part of codename “Oslo” – the modeling capabilities, the repository and MGrammar – to create custom DSLs in our business. We try to be early adopters of a lot of the new Microsoft technologies, but we’re not about to try to push things into production without a “Go-Live” license.

    Q [stupid question]: This past year you received your first Microsoft MVP designation for your work in Connected Systems.  There are a surprising number of technologies that have MVPs, but they could always use a few more such as a Notepad MVP, Vista Start Menu MVP or Microsoft Word “About Box” MVP.  Give me a few obscure/silly MVP possibilities that Microsoft could add to the fold.

    A: Well, I’ve seen a lot of middle-aged++ people during my career that could easily fit into a “Solitaire MVP” category 🙂 Fun aside, I’m a bit curious why Microsoft have Zune & XBox MVP titles. Last time I checked, the P was for “Professional” I can hardly imagine anyone who gets paid for listening to their Zune or playing on their XBox. Now, I don’t mean  to offend the Zune & XBox MVPs, because I know they’re brilliant at what they do, but Microsoft should probably have a different badge to award people that are great at leisure activities, that’s all.

    Thanks Lars for a good chat.

  • Win a Free Copy of my Book

    So Stephen Thomas is running a little competition to give away a copy of my recent book.  Stephen calls out the details in a blog post, but basically, he’s looking for a great BizTalk tip or trick, and will award the book to the best submission.  You’ve got a week to respond.

    Stephen is running a very appropriate contest for a book like this.  It’s a good thing that HE’S doing this and not me, since my contest would something more like “whoever can first take a picture of themselves shaving a panda” or “whoever sends me the largest body part via FedEx overnight shipping.”  So thank yourselves for small favors.

  • Tidbits from Amazon.com AWS Cloud Briefing

    Yesterday I attended the first of a set of roving sessions from Amazon.com to explain their cloud offering, Amazon Web Services (AWS).  I’ve been tinkering with their stuff for a while now, but I was amped to hear a bit more from the horse’s mouth.  I went with a couple colleagues and greatly enjoyed the horrific drive to/from Beverly Hills.

    The half-day session was keynoted by Amazon.com CTO Dr. Werner Vogels.  He explained how Amazon’s cloud offering came to be, and how it helped them do their jobs better.  He made a number of good points during his talk:

    • We have to think of reality scale vs. an academic or theoretical model of how an application scales.  That is, design scalability for real life and your real components and don’t get caught up in academic debates on how a system SHOULD scale.
    • He baked a key aspect of SOA down to “the description of a service is enough to consume it.”  That’s a neat way to think about services.  No libraries required, just standard protocols and a definition file.
    • If your IT team has to become experts in base functions like high performance computing in order to solve a business problem, then you’re not doing IT right and you’re wasting time.  We need to leverage the core competencies (and offerings) of others.
    • Amazon.com noticed that when it takes a long time (even 6 hours) to provision new servers, then people are more hesitant to release the resource when they were done with it.  This leads to all sorts of waste and inefficiency, and this behavior can be eliminated by an on-demand, pay-as-you-go cloud model.
    • Amazon.com breaks down the application into core features and services to the point that each page leverages up to 300 distinct services.  I can’t comprehend that without the aid of alcohol.
    • We need to talk to our major software vendors about cloud-driven licensing.  Not the vendor’s cloud computing solution, but how I can license their software in a cloud environment where I may temporarily stand up servers.  Should I pay a full CPU license for a database if I’m only using it during cloud-bursting scenarios or for short-lived production apps, or should I have a rental license available to me?

    Werner gave a number of good case study mentions ranging from startups to established, mature organizations.  Everything from eHarmony.com using the parallel processing of Amazon MapReduce to do “profile matching” in the cloud, to a company like SAP putting source code in the cloud in the evenings and having regresssion tests run against it.  I was amused by the eHarmony.com example only because I would hate to be the registered member who finally made the company say “look, we simply need 1400 simultaneously running computers to find this gargoyle a decent female companion.” 

    Representatives from Skifta, eHarmony.com, Reddit, and Geodelic were all on hand to explain how they used the Amazon cloud and what they learned while leveraging it.  Good lessons around working around unavoidable latency, distributing data, and sizing on demand.

    The session closed with talks from Mike Culver and Steve Riley of AWS.  Mike talked about architectural considerations (e.g. design for failure, force loose coupling, design for elasticity, put security everywhere, consider the best storage option) while Steve (a former Microsoftie) talked about security considerations.  My boss astutely noticed that most (all?) of Mike’s points should pertain to ANY good architecture, not necessarily just cloud.  Steve talked a fair amount about the AWS Virtual Private Cloud which is a pretty slick way to use the Amazon cloud, but put these machines within the boundaries (IP, domain, management) of your own network.

    All in all, great use of time.  We thought of additional use cases for my own company including proof-of-concept environments, providing temporary web servers for new product launches, or trying to process our mounds of drug research data more efficiently.

    If you get the chance to attend the upcoming New York or London sessions, I highly recommend it.

    Share

  • Orchestrating the Cloud: Part II – Creating and Consuming a Salesforce.com Service From BizTalk Server

    In my previous post, I explained my cloud orchestration scenario where my on-premises ESB coordinated calls to the Google App Engine, Salesforce.com and a local service, and returned a single data entity to a caller.  That post looked at creating and consuming a Google App Engine service from BizTalk.

    In this post, I’ll show you how to customize a data object in Force.com, expose that object via a web service, and invoke that from BizTalk Server.  As a ridiculously short primer, SalesForce.com is considered the premier SaaS CRM product which provides sales force automation and customer service modules serving both large and small organizations alike.  Underlying SalesForce.com is a scalable robust platform (Force.com) which can be used to build all sorts of data-driven applications. You can leverage the apps built by others through the AppExchange which lists a diverse range of applications built on Force.com.

    Ok, enough of a sales job.  First off, I signed up for a free Force.com account.  I’m going to extend the out-of-the-box “Contact” object by adding a few new fields.  The “Setup” section of my Force.com application provides me access to a host of options to create new things, customize existing things and turn all sorts of knobs that enable rich functionality.  Here I browsed to the “Contact” object and chose “Fields”.

    2009.10.07force01

    Next I created a few custom fields to hold a global identifier (across all my CRM applications), a contact preference and a list of technology interests of the contact.

    2009.10.07force02

    I then navigate to my “Contacts” page and see my custom fields on the screen.  I can move then anywhere on the screen that I like using an easy-to-use drag-and-drop interface.

    2009.10.07force03

    Now that my data object is complete, I want to create a web service that lets my on-premises ESB retrieve customers based on their Global ID.  Back within the Force.com “Setup” screens I chose to Develop a new Apex class.  Note that Apex is the C#/Java-like language used to write code for Force.com.

    2009.10.07force04

    My class, named CRMCustomer has a web service operation identified where I lookup the contact with the ID matching the service’s input parameter, and then deliver a subset of the full Contact object back to the service caller.  If you look closely you can see that some fields have a “__c” after the field name to designate them as custom.

    2009.10.07force05

    If my class is written successfully, I’ll see an entry in my list of classes.  Note that my class now has a “WSDL” link next to it.

    2009.10.07force06

    Ok, now I have the object and service that I need for BizTalk to call this Force.com service.  But, I still need to retrieve my service definition.  First, I clicked this WSDL link next to my Apex class and saved the WSDL to my BizTalk machine.  Every time that I call the Force.com service I need to pass an access token in the header.  The header definition can be found in the Enterprise WSDL, which I also saved to my BizTalk machine.

    2009.10.07force07

    I made a choice to cache the temporary Force.com access token so that each call to my custom service wouldn’t have to do two invocations.  I accomplished this by building a singleton class which expires its token and reacquires a new one every hour.  That class library project has a reference to the Salesforce.com Enterprise WSDL.

    [Serializable]
        public static class ForceToken
        {
            private static DateTime _sessionDate = DateTime.Now;
            private static string _sessionId = string.Empty;
            public static string SessionId
            {
                get { return GetSession(); }
            }
            private static string GetSession()
            {
                DateTime now = DateTime.Now;
                TimeSpan diff = now.Subtract(_sessionDate);
                if (_sessionId == string.Empty || (diff.TotalMinutes > 60))
                {
                    //TODO lock object during update
                    //refresh token
                    System.Diagnostics.EventLog.WriteEntry("Utilities", "Salesforce.com Session Refresh");
                    string uname = "<sf account>";
                    string password = "<sf password>";
                    string securityToken = "<sf token>";
                    SFSvcRef.SforceService proxy = new SFSvcRef.SforceService();
                    proxy.Url = "https://www.salesforce.com/services/Soap/c/16.0";
                    SFSvcRef.LoginResult result = proxy.login(uname, password + securityToken);
                    _sessionId = result.sessionId;
                }
                return _sessionId;
            }
        }

    Within my actual BizTalk project, I added a service reference to the Force.com custom WSDL that was saved to my machine.  Lots of things come in, including the definition of the session header and my modified Contact object.

    2009.10.07force08

    Notice that the response object holds my custom fields such as “Contact Preference.”

    2009.10.07force09

    I’m using an orchestration to first get the access token from my singleton, and then put that token into the WCF header of the outbound message.

    2009.10.07force10

    Inside the Assignment shape is the simple statement that populates the SOAP header of my Force.com service call.

    CRMCustomer_Request(WCF.Headers) = "<headers><SessionHeader><sessionId>"+ Seroter.SwedenUG.Utilities.ForceToken.SessionId +"</sessionId></SessionHeader></headers>";

    My send port was created automatically from the binding file produced when importing the Force.com custom WSDL.  This WCF-Custom send port uses the basicHttp binding to call the endpoint.

    2009.10.07force12

    Once I send a message to my orchestration which contains the “global ID” of the record that I’m looking for, the Force.com service is called and my record is returned.

    2009.10.07force11

    Cool.  That’s a live record in my Force.com application (shown in a screenshot earlier) and can be pulled on-demand via my service.

    So what we know now?

    • Easy to set up a Force.com account
    • There is a straightforward interface to customize objects and build web services
    • BizTalk needs to request a time-limited token for it service calls so a singleton can introduce some efficiency
    • You can add the session header to the outbound message via a WCF context property accessor in an orchestration

    Next up, I’ll show how I tie all this together with an web application hosted in Amazon.com’s EC2 environment and leveraging the Azure .NET Service Bus to communicate between Amazon’s public cloud and my on-premise ESB.

    Share

  • Orchestrating the Cloud : Part I – Creating and Consuming A Google App Engine Service From BizTalk Server

    I recently wrote about my trip to Stockholm where I demonstrated some scenarios showing how I could leverage my onsite ESB in a cloud-focused solution.  The first scenario I demonstrated was using BizTalk Server 2009 to call a series of cloud services and return the result of that orchestrated execution back to a web application hosted in the Amazon.com EC2 cloud.  This series of blog posts will show how I put each piece of this particular demonstration together.

    2009_09_21cloud01

    In this first post, I’ll show how I created a Python web application in the Google App Engine which allows me to both add/delete data via a web UI and provides a POX web service for querying data.  I’ll then call this application from BizTalk Server to extract relevant data.

    As you’d expect, the initial step was to build the Google App Engine web app.  First, you need to sign up for a (free) Google App Engine account.  Then, if you’re like me and building a Python app (vs. Java) you can go here and yank all the necessary SDKs.  You get a local version of the development sandbox so that you can fully test your application before deploying it to the Google cloud.

    Let’s walk through the code I built.  As a disclaimer, I learned Python solely for this exercise, and I’m sure that my code reflects the language maturity of a fetus.  Whatever, it works.  Don’t judge me.  But either way, note that there are probably better ways to do what I’ve done, but I couldn’t find them.

    First off, I have some import statements to libraries I’ll use within my code.

    import cgi
    from google.appengine.ext import webapp
    from google.appengine.ext.webapp.util import run_wsgi_app
    from google.appengine.ext import db
    from xml.dom import minidom
    from xml.sax.saxutils import unescape

    Next I defined a “customer” object which represents the data I wish to stash in the Datastore.

    #customer object definition
    class Customer(db.Model):
        userid = db.StringProperty()
        firstname = db.StringProperty()
        lastname = db.StringProperty()
        currentbeta = db.StringProperty()
        betastatus = db.StringProperty()
        dateregistered = db.StringProperty()

    At this point, I’m ready for the primary class which is responsible for drawing the HTML page where I can add/delete new records to my application. First I define the class and write out the header of the page.

    #main class
    class MainPage(webapp.RequestHandler):
        def get(self):
            #header HTML
            self.response.out.write('<html><head><title>Vandelay Industries Beta Signup Application</title>')
            self.response.out.write('<link type=\"text/css\" rel=\"stylesheet\" href=\"stylesheets/appengine.css\" /></head>')

            self.response.out.write('<body>')
            self.response.out.write('<table class=\"masterTable\">')

            self.response.out.write('<tr><td rowspan=2><img src=\"images/vandsmall.png\"></td>')

            self.response.out.write('<td class=\"appTitle\">Beta Technology Sign Up Application</td></tr>')

            self.response.out.write('<tr><td class=\"poweredBy\">Powered by Google App Engine<img src=\"images/appengine_small.gif\"></td></tr>')

    Now I want to show any existing customers stored in my system.  Before I do my Data Store query, I write the table header.

    #show existing customer section
            self.response.out.write('<tr><td colspan=2>')
            self.response.out.write('<hr width=\"75%\" align=\"left\">')

            self.response.out.write('<span class=\"sectionHeader\">Customer List</span>')

            self.response.out.write('<hr width=\"75%\" align=\"left\">')

            self.response.out.write('<table class=\"customerListTable\">')

            self.response.out.write('<tr>')
            self.response.out.write('<td class=\"customerListHeader\">ID</td>')

            self.response.out.write('<td class=\"customerListHeader\">First Name</td>')

            self.response.out.write('<td class=\"customerListHeader\">Last Name</td>')

            self.response.out.write('<td class=\"customerListHeader\">Current Beta</td>')

            self.response.out.write('<td class=\"customerListHeader\">Beta Status</td>')

            self.response.out.write('<td class=\"customerListHeader\">Date Registered</td>')

            self.response.out.write('</tr>')

    Here’s the good stuff.  Relatively.  I query the Datastore using a SQL-like syntax called GQL and then loop through the results and print each returned record.

    #query customers from database
           customers = db.GqlQuery('SELECT * FROM Customer')
           #add each customer to page
           for customer in customers:
               self.response.out.write('<tr>')
               self.response.out.write('<td class=\"customerListCell\">%s</td>' % customer.userid)

               self.response.out.write('<td class=\"customerListCell\">%s</td>' % customer.firstname)

               self.response.out.write('<td class=\"customerListCell\">%s</td>' % customer.lastname)

               self.response.out.write('<td class=\"customerListCell\">%s</td>' % customer.currentbeta)

               self.response.out.write('<td class=\"customerListCell\">%s</td>' % customer.betastatus)

               self.response.out.write('<td class=\"customerListCell\">%s</td>' % customer.dateregistered)

               self.response.out.write('</tr>')
           self.response.out.write('</table><br/><br />')
           self.response.out.write('</td></tr>')

    I then need a way to add new records to the application, so here’s a block that defines the HTML form and input fields that capture a new customer.  Note that my form’s “action” is is set to “/Add”.

    #add customer entry section
            self.response.out.write('<tr><td colspan=2>')
            self.response.out.write('<hr width=\"75%\" align=\"left\">')

            self.response.out.write('<span class=\"sectionHeader\">Add New Customer</span>')

            self.response.out.write('<hr width=\"75%\" align=\"left\">')

            self.response.out.write('<form action="/Add" method="post">')
            self.response.out.write('<table class=\"customerAddTable\">')

            self.response.out.write('<tr><td class=\"customerAddHeader\">ID:</td>')

            self.response.out.write('<td class=\"customerListCell\"><input type="text" name="userid"></td></tr>')

            self.response.out.write('<tr><td class=\"customerAddHeader\">First Name:</td>')

            self.response.out.write('<td class=\"customerListCell\"><input type="text" name="firstname"></td></tr>')

            self.response.out.write('<tr><td class=\"customerAddHeader\">Last Name:</td>')

            self.response.out.write('<td class=\"customerListCell\"><input type="text" name="lastname"></td></tr>')

            self.response.out.write('<tr><td class=\"customerAddHeader\">Current Beta:</td>')

            self.response.out.write('<td class=\"customerListCell\"><input type="text" name="currentbeta"></td></tr>')

            self.response.out.write('<tr><td class=\"customerAddHeader\">Beta Status:</td>')

            self.response.out.write('<td class=\"customerListCell\"><input type="text" name="betastatus"></td></tr>')

            self.response.out.write('<tr><td class=\"customerAddHeader\">Date Registered:</td>')

            self.response.out.write('<td class=\"customerListCell\"><input type="text" name="dateregistered"></td></tr>')

            self.response.out.write('</table>')
            self.response.out.write('<input type="submit" value="Add Customer">')
            self.response.out.write('</form><br/>')
            self.response.out.write('</td></tr>')

    Finally, I have an HTML form for a delete behavior which has an action of “/Delete.”

    #delete all section
            self.response.out.write('<tr><td colspan=2>')
            self.response.out.write('<hr width=\"75%\" align=\"left\">')

            self.response.out.write('<span class=\"sectionHeader\">Delete All Customer</span>')

            self.response.out.write('<hr width=\"75%\" align=\"left\">')

            self.response.out.write('<form action="/Delete" method="post"><div><input type="submit" value="Delete All Customers"></div></form>')
            self.response.out.write('</td></tr>')
            self.response.out.write('</table>')
            #self.response.out.write('')
            #write footer
            self.response.out.write('</body></html>')

    The bottom of my “.py” file has the necessary setup declarations to fire up my default class and register behaviors.

    #setup
    application = webapp.WSGIApplication([('/', MainPage)],debug=True)
    def main():
        run_wsgi_app(application)
    if __name__ == "__main__":
        main()

    If I open a DOS prompt, navigate to the parent folder of my solution (and assuming I have a valid app.yaml file that points at my .py file), I can run the dev_appserver.py serotercustomer/ command and see a local, running instance of my web app.

    2009.10.01gae01

    Cool.  Of course I still need to wire the events up for adding, deleting and getting a customer.  For the “Add” operation, I create a new “customer” object, and populate it with values from the form submitted on the default page.  After calling the “put” operation on the object (which adds it to the Datastore), I jump back to the default HTML page.

    #add customer action class
    class AddCustomer(webapp.RequestHandler):
        def post(self):
            customer = Customer()
            customer.firstname = self.request.get('firstname')
            customer.lastname = self.request.get('lastname')
            customer.userid = self.request.get('userid')
            customer.currentbeta = self.request.get('currentbeta')
            customer.betastatus = self.request.get('betastatus')
            customer.dateregistered = self.request.get('dateregistered')
            #store customer
            customer.put()
            self.redirect('/')

    My “Delete” is pretty coarse as all it does is delete every customer object from the Datastore.

    #delete customer action class
    class DeleteCustomer(webapp.RequestHandler):
        def post(self):
            customers = db.GqlQuery('SELECT * FROM Customer')
            for customer in customers:
                customer.delete()
            self.redirect('/')

    The “Get” operation is where I earn my paycheck.  This “Get” is called via a system (i.e. not the user interface) so it needs to accept XML in, and return XML back.  So what I do is take the XML I received into the HTTP POST command, unescape it, load it into an XML DOM, and pull out the “customer ID” node value.  I then execute some GQL using that customer ID and retrieve the corresponding record from the Datastore.  I inflate an XML string, load it back into a DOM object, and return that to the caller.

    #get customer action class
    class GetCustomer(webapp.RequestHandler):
        def post(self):
            #read inbound xml
            xmlstring = self.request.body
            #unescape to XML
            xmlstring2 = unescape(xmlstring)
            #load into XML DOM
            xmldoc = minidom.parseString(xmlstring)
            #yank out value
            idnode = xmldoc.getElementsByTagName("userid")
            userid = idnode[0].firstChild.nodeValue
            #find customer
            customers = db.GqlQuery('SELECT * FROM Customer WHERE userid=:1', userid)
            customer = customers.get()
            lastname = customer.lastname
            firstname = customer.firstname
            currentbeta = customer.currentbeta
            betastatus = customer.betastatus
            dateregistered = customer.dateregistered
            #build result
            responsestring = """"" % (userid, firstname, lastname, currentbeta, betastatus, dateregistered)
            <CustomerDetails>
                <ID>%s</ID>
                <FirstName>%s</FirstName>
                <LastName>%s</LastName>
                <CurrentBeta>%s</CurrentBeta>
                <BetaStatus>%s</BetaStatus>
                <DateRegistered>%s</DateRegistered>
            </CustomerDetails>
            "

            #parse result
            xmlresponse = minidom.parseString(responsestring)
            self.response.headers['Content-type'] = 'text/xml'
            #return result
            self.response.out.write(xmlresponse.toxml())

    Before running the solution again, I need to update my “setup” statement to register the new commands (“/Add”, “/Delete”, “/Get”).

    #setup
    application = webapp.WSGIApplication([('/', MainPage),
                                          ('/Add', AddCustomer),
                                          ('/Delete', DeleteCustomer),
                                          ('/Get', GetCustomer)],
                                         debug=True)

    Coolio.  If I run my web application now, I can add and delete records and any records in the store show up in the page.  Now I can deploy my app to the Google cloud using the the console or the new deployment application.  I then added a few sample records that I could use BizTalk to lookup later.

    2009.10.01gae05

    The final thing to do is have BizTalk call my POX web service.  In my new BizTalk project, I built a schema for the service request.  Remember that all it needs to contain is a customer ID.  Also note that my Google App Engine XML is simplistic and contains no namespaces.  That’s no problem for a BizTalk schema.  Neither of my hand-built Google App Engine XSDs have namespaces defined.  Here is my service request schema:

    2009.10.01gae02

    The POX service response schema reflects the XML structure that my service returns.

    2009.10.01gae03

    Now that I have this, I decided to use a solicit-response BizTalk HTTP adapter to invoke my service.  The URL of my service was: http://<my app name>.appspot.com/Get which leverages the “Get” operation that will accepts the HTTP post request.

    Since I don’t have an orchestration yet, I can just use a messaging scenario and have a FILE send port that subscribes on the response from the solicit-response HTTP port.  When I send in a file with a valid customer ID, I end up with a full response back from my POX web service.

    2009.10.01gae04

    So there you go.  Creating a POX web service in the Google App Engine and using BizTalk Server to call it.  Next up, using BizTalk to extract data from a SalesForce.com instance.

    Share

  • Interview Series: Four Questions With … Jan Eliasen

    We’ve gotten past the one-year hump of this interview series, so you know that I’m in it for the long haul.  I’m sure that by the end of the year I’ll be interviewing my dog, but for now, I still have a whole stable of participants on my wish list.  One of those was Jan Eliasen.  He’s a 3-time Microsoft MVP, great blogger, helpful BizTalk support forum guy, and the pride of Denmark.  Let’s see how he makes it through four questions with me.

    Q: You’ve recently announced on your blog that you’ll be authoring a BizTalk book alongside some of the biggest names in the community.  What’s the angle of this book that differentiates it from other BizTalk books on the market?  What attracted you personally to the idea of writing a book?

    A: Well, first of all, this will be my first book and I am really excited about it. What attracted me, then? Well, definitely not the money! 🙂  I am just really looking forward to seeing my name on a book. It is a goal that I didn’t even know I had before the publisher contacted me :). Our book will be written by 7 fantastic people (if everyone signs the contract 🙂 ), each with extremely good knowledge of BizTalk and naturally, we have divided the content amongst us to accommodate our particular expertise, meaning that each area of the book will be written by someone who really knows what he is talking about (Charles Young on BRE, Jon Flanders on WCF, Brian Loesgen on ESB and so on). We will be focusing on how to solve real and common problems, so we will not go into all the nitty gritty details, but stay tuned to what is needed by developers to develop their solutions.

    Q: Messaging-based BizTalk solutions are a key aspect of any mature BizTalk environment and you’ve authored a couple of libraries for pipelines and mapping functoids to help out developers that build solutions that leverage BizTalk’s messaging layer.  Do you find that many of the solutions you build are “messaging only”?  How do you decide to put a capability into a map or pipeline instead of an orchestration?

    A: My first priority is to solve problems using the architecture that BizTalk provides and wants us to use – unless I have some performance requirements that I can’t meet and that therefore forces me to “bend” the architecture a bit.

    This means, that I use pipeline components to do what they are meant to – processing of a message before it is published. Take my “Promote” component for instance. It is used to promote the value given by an XPath expression (or a constant) into a promoted property. This enables you to promote the value of a reoccurring element, but the developer needs to guarantee that one and only one value is the result of this XPath. Since promoted properties are used for routing, it makes sense to do this in a pipeline (that all ready does most of the promoting anyway) because they need to be promoted before publishing in order to be useful.

    But really, my point is, that unless forced to do so, I don’t compromise with the architecture. You will not find me writing a .NET assembly for inserting rows into SQL Server instead of using the SQL adapter or use custom XSLT instead of functoids if the functoids can do the job – or other such things unless I am really forced to do so. My thoughts are, that if I stay within the built in functionality, it will be easier for anyone to understand my solution and work on it after I am gone 🙂

    Q: As the products and tools for building web service and integration solutions get better and better, it can shift the source of complexity from one area to another.  What is an example of something that is easier to do than people think, and an example of something that is actually harder than people think.

    A: Well, one thing that is easier than people think is to program your own functoids and pipeline components. Functoids are actually quite easy – the difficult part is getting a reasonable icon done in 16×16 pixels 🙂 If you look at my functoids, you will see that I am not to be trusted with an imaging program. Pipeline components, also – much easier than people think… for the simple ones anyway – if performance gets an issue, you will need the streaming way of doing things, which will add some complexity, but still… most people get scared when you tell them that they should write their own pipeline component… People really shouldn’t 🙂

    Thing that is harder than people think: Modeling your business processes. It is really hard to get business people to describe their processes – there are always lots of “then we might do this or we might do that, depending on how wet I got last week in the rain.” And error handling, especially – getting people to describe unambiguously what to be done in case of errors – which often to users involve “Then just undo what you did” without considering that this might not be possible to automate – I mean… if I call a service from an ERP system to create an order and I later on need to compensate for this, then two things must be in place: 1: I need an ID back from the “CreateOrder” service and 2: I need a service to call with that ID. Almost always; This ID and this service do not exist. And contrary to popular belief, neither BizTalk nor I can do magic 🙂

    Q [stupid question]: I recently had a long plane flight and was once again confronted by the person in front of me pushing their seat all the way back and putting their head in my lap.  I usually get my revenge by redirecting my air vent to blow directly on the offender’s head so that if they don’t move their seat up, they at least get a cold.  It’s the little things that entertain me.  How do you deal with unruly travelers and keep yourself sane on plane/train trips?

    A: So, for keeping sane, naturally, refer to http://tribes.tribe.net/rawadvice/thread/7cb09c39-efa1-415e-9e84-43e44e615cae – there are some good advice for everyone 🙂 Other than that, if you just dress in black, shave your head, have a couple of piercings and wear army boots (Yes, I was once one of “those”….) no one even thinks about putting their head in your lap 🙂

    Great answers, Jan.  Hope you readers are still enjoying these monthly chats.

    Share