Richard Seroter's Architecture Musings

Author: Richard Seroter

  • My BizTalk Code Review Checklist

    I recently put together a BizTalk Code Review checklist for our development teams, and thought I’d share the results.

    We didn’t want some gargantuan list of questions that made code review prohibitive and grueling. Instead, we wanted a collection of common sense, but concrete, guidelines for what a BizTalk solution should look like. I submit that any decent BizTalk code reviewer would already know to look out for the items below, but, having the checklist in written form ensures that developers starting new projects know EXACTLY what’s expected of them.

    I’m sure that I’ve missed a few things, and would welcome any substantive points that I’ve missed.

    BizTalk Code Review Checklist

    Naming Standards Review
    Standard Result Correction Details
    Pass Fail
    Visual Studio.NET solution name follows convention of:
    [Company].[Dept].[Project]
    Visual Studio.NET project name follows convention of:
    [Company].[Dept].[Project].[Function]
    Schema name follows convention of:
    [RootNodeName]_[Format].xsd
    Property schema name follows convention of:
    [DescriptiveName]_PropSchema.xsd
    XSLT map name follows convention of:
    [Source Schema]_To_[Dest Schema].btm
    Orchestration name follows convention of:
    [Meaningful name with verb-noun pattern].odx
    Pipeline name follows convention of:
    Rcv_[Description].btp /
    Snd_[Description].btp
    Orchestration shape names match BizTalk Naming Standards document
    Receive port name follow convention of:
    [ApplicationName].Receive[Description]
    Receive location name follows convention of:
    [Receive port name].[Transport]
    Send port name follows convention of:
    [ApplicationName].Send[Description].[Transport]

    Schema Review
    Standard Result Correction Details
    Pass Fail
    Namespace choice consistent across schemas in project/name
    Nodes have appropriate data types selected
    Nodes have restrictions in place (e.g. field length, pattern matching)
    Nodes have proper maxOccurs and minOccurs values
    Node names are specific to function and clearly identify their contents
    Auto-generated schemas (via adapters) have descriptive file names and “types”
    Schemas are imported from other locations where appropriate to prevent duplication
    Schemas that import other schemas have a “root reference” explicitly set
    Clear reasons exist for the values promoted in the schema
    Schema elements are distinguished appropriately
    Schema successfully “validates” in Visual Studio.NET
    Multiple different instance files successfully validate against the schema

    Mapping Review
    Standard Result Correction Details
    Pass Fail
    Destination schema has ALL elements defined with either an inbound link, functoid, or value.
    Functoids are used correctly
    Scripting functoid has limited inline code or XSLT.
    Scripting functoid with inline code or XSLT is well commented
    Database functoids are not used
    Multiple “pages” are set up for complex maps
    Conversion between data types is done in functoids (where necessary)
    Map can be validated with no errors
    Multiple different input instance files successfully validate against the map

    Orchestration Review
    Standard Result Correction Details
    Pass Fail
    Each message and variable defined in the orchestration are used by the process
    Transactions are used appropriately
    All calls to external components are wrapped in an exception-handling Scope
    No Expression shape contains an excessive amount of code that could alternately be included in an external component
    The Parallel shape is used correctly
    The Listen shape is not used in place of transaction timeouts
    All Loops have clearly defined exit conditions
    Where possible, message transformations are done at the “edges” (i.e. port configurations)
    Calling one orchestration from another orchestration is done in a manner that supports upgrades
    Correlation is configured appropriately
    All messages are created in an efficient manner
    The message is not “opened” in unnecessary locations
    All variables are explicitly instantiated
    No port operations are named the default “Operation_1”
    Port Types are reused where possible
    All Request/Response ports exposed as a web service are equipped with a SOAP fault message.
    Orchestration has trace points inserted to enable debugging in later environments
    Orchestration design patterns are used wherever possible

    Business Rule Review
    Standard Result Correction Details
    Pass Fail
    Business rule output tested for all variations of input
    Conflict resolution scenarios are non-existent or limited
    Long-term fact retrievers used for static facts
    Business Rule vocabulary defined for complex rule sets

    Configuration Review
    Standard Result Correction Details
    Pass Fail
    Receive Port / Send Port tracking configurations appropriately set
    Maps are applied on the Receive Port where appropriate
    Send port retry interval set according to use case
    Maps are applied on Send Port where appropriate
    Send port does NOT have filter attached if connected to an orchestration
    Subscriptions exist for every message processed by the application

    Deployment Package Review
    Standard Result Correction Details
    Pass Fail
    “Destination Location” for each artifact uses “%BTAD_InstallDir%” token vs. hard coded file path
    All supporting artifacts (e.g. helper components, web services, configuration files) are added as Resources
    Binding file is NOT a resource if ports use transports with passwords

    Overall Solution Architecture Review
    Standard Result Correction Details
    Pass Fail
    Solution is organized in Visual Studio.NET and on disk in a standard fashion
    Passwords are never stored in clear text
    All references to explicit file paths are removed / minimized
    All two-way services INTO BizTalk produce a response (either expected acknowledgement or controlled exception message)
    Calls to request/response web services that take an exceptional amount of time to process are reengineered to use an “asynchronous callback” pattern
    Exceptions are logged to an agreed upon location
    Long-running processes have a way to inspect progress to date
    Solution has been successfully tested with REAL data from source systems
    Solution has been successfully tested while running under user accounts with permissions identical to the production environment
    Messages are validated against their schema per use case requirements
    Processes are designed to be loosely coupled and promote reuse where possible

    Technorati Tags:

  • Important Hotfixes For the BizTalk Oracle Adapter

    We’ve encountered a few quirky things with the Oracle database adapter for BizTalk, so I thought I’d point out a few Microsoft KB articles and hotfixes that you should be aware of if you’re using this adapter.

    For some reason you need a compass and secret handshake to find these freakin’ things on the Microsoft website, so to grab the full list of Oracle adapter KB articles, visit here.

    Technorati Tags:

  • Interesting BizTalk Links From This Week

    I usually despise blogs that do nothing but link to other posts (breaking rule #11 in Jeff’s list of blog clichés), but, there were a few BizTalk/service things that caught my eye this week and I thought I’d briefly point them out.

    Good stuff.

    Technorati Tags:

  • Debatching Flat Files Into XML (With Header Intact)

    Today I was approached with the problem of how to take a flat file input, debatch it, add the “header” to each individual record, and return the results as XML messages.

    The first part of the equation was to successfully debatch the flat file. So, I took the header and body schemas that the developer built and added them to a new Receive pipeline’s “flat file disassembler”. On the flat file disassembler, I flipped the Preserve header property to “True” to ensure that the header would be injected into the context collection of each debatched record.

    I can prove that Preserve header works by stopping my send port, and checking out the temporarily suspended messages. If I view the context collection for a given message, I’ll see the FlatFileHeaderDocument value and my header message (in XML format) embedded inside.

    To transmit a flat file result (with header attached), all I have to do is create a custom Send pipeline, use the flat file assembler component, and once again choose the document and header schemas. This will cause the FlatFileHeaderDocument to be sucked out of context and converted back to delimited text, and slapped on top of each message.

    That’s cool, but in our case, we wanted XML output with the header appended. I could have tried some gnarly solution involving orchestrations calling a “flat file” send pipeline (to add the header) and then an “xml” send pipeline to generate XML content, but that didn’t seem clean. Instead, I created an orchestration that first grabs the FlatFileHeaderDocument out of context and creates a valid orchestration message from it.

    The XMLNORM.FlatFileHeaderDocument is the specific context value from which I can yank a string representation of the XML header message. I then loaded that into an XmlDocument variable and used that variable to instantiate an orchestration message. Next, I passed both the body AND the newly-constructed header message into a multi-part map. This map takes those two values and outputs a single document format containing both header and body values.

    So now, BizTalk can take the flat file containing 20 records, debatch it, and reassemble it (header and all) into valid XML output.

    Technorati Tags:

  • Behavior Of Static Objects In BizTalk Runtime

    I recently answered a BizTalk newsgroup post where the fellow was asking how static objects would be shared amongst BizTalk components. I stated that a correctly built singleton should be available to all artifacts in the host’s AppDomain. However, I wasn’t 1000% sure what that looked like, so I had to build out an example.

    When I say a “correctly built singleton”, I mean a thread-safe static object. My particular singleton for this example looks like this:

    public class CommonLogger
    {
        //static members are lazily initialized, but thread-safe
        private static readonly CommonLogger singleton =
            new CommonLogger();
        private int Id;
        private string appDomainName;

        //Explicit static constructor 
        static CommonLogger() { }
        private CommonLogger() 
        {
            appDomainName = AppDomain.CurrentDomain.FriendlyName;

            System.Random r = new Random();
            //set "unique" id
            Id = r.Next(0, 100);

            //trace
            System.Diagnostics.Debug.WriteLine
	("[AppDomain: " + appDomainName + ", ID: " + 
                Id.ToString() + "] Logger started up ... ");
        }

        //Accessor
        public static CommonLogger Instance
        {
            get
            {
                return singleton;
            }
        }

        public void LogMessage(string msg)
        {
            System.Diagnostics.Debug.WriteLine
	("[AppDomain: " + appDomainName + "; ID: " + 
                  Id.ToString() + "] Message logged ... " + msg);
        }
    }

    I also built a “wrapper” class which retrieves the “Instance” object for BizTalk artifacts that couldn’t access the Instance directly (e.g. maps, orchestration).

    Next, I built a custom pipeline component (send or receive) where the “Execute” operation makes a call to my CommonLogger component. That code is fairly straightforward and looks like this …

    public IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
        {
            //call singleton logger
            CommonLogger.Instance.LogMessage
               ("calling from " + _PipelineType + " component");

            // this way, it's a passthrough pipeline component
            return inmsg;
        }

    Next, I created a simple map containing a Scripting functoid that called out to my CommonLogger. Because the Scripting functoid can’t call an operation on my Instance property, I used the “Wrapper” class which executes the operation on the Instance.

    Then I went ahead and created both a send and receive pipeline, each with my custom “logging” component built in. After deploying the BizTalk projects, I added my map and pipeline to both the receive and send port. So, there are four places that should be interacting with my CommonLogger object. What’s the expected result of this run? I should see “Logger started up” (constructor) once, and then a bunch of logged messages using the same instance. Sure enough …

    Nice. What happens if I throw orchestration into the mix? Would it share the object instantiated by the End Point Manager (EPM)? If you read Saravana’s post, you get the impression that every object in a single host should share an AppDomain, and thus static objects. I wasn’t convinced that this was the case.

    I’ve also added a business rule to the equation, to see how that plays with the orchestration.

    I bounced my host instance (thus flushing any cached objects), and reran my initial scenario (with orchestration/rules included):

    Very interesting. The orchestration ran in a different AppDomain, and thus created its own CommonLogger instance. Given that XLANG is a separate subsystem within the BizTalk service, its not impossible to believe that it runs in a separate AppDomain.

    If I run my scenario again, without bouncing the host instance, I would expect to see no new instantiations, and objects being reused. Image below is the same as the previous one (first run), with the subsequent run in the same window.

    Indeed, the maps/pipeline reused their singleton instance, and the orchestration/rules reused their particular instance. Now you can read all about creating your own named AppDomains for orchestrations in this MSDN documentation, and maybe, because I don’t have a named instance for this orchestration to run in, an ad-hoc one is being created and used. Either way, it seems that the EPM and XLANG subsystems run in different AppDomains within a given host instance.

    I also experimented with moving my artifacts into different hosts. In this scenario, I moved my send port out of the shared host, and into a new host. Given that we’re now in an entirely different Windows service, I’d hardly expect any object sharing. Sure enough …

    As you can see, now three different instances of my singleton object exist and are actively being used within their given AppDomain. Does this matter much? In most cases, not really. I’m still getting the value of caching and using a thread-safe singleton object. There just happen to be more than one instance being used by the BizTalk subsystems. That doesn’t negate the value of the pattern. But, still valuable to know.

    Technorati Tags:

  • BizTalk Property Schemas Separated From Associated Schemas, Take II

    Back on the old Microsoft blog, I wrote about not separating a property schema from its implementation schema. I concluded that this “tip” in the documentation seemed to be more like a guideline vs. a rule.

    Today, I realized it’s more like a rule. I had a BizTalk project containing ONLY a property schema, then a BizTalk project ONLY containing the schemas that reference the property schema, and finally a BizTalk project containing an orchestration that used the various schemas. For the life of me, I couldn’t figure out why my promoted fields wouldn’t show up in the Receive shape’s filter expression, or, as part of the message (by doing “myMessage(Namespace.PropSchemaValue) = 1234). Funny enough, the property schema value marked as MessageContextPropertyBase DID show up, but any of the MessageDataPropertyBase were noticeably absent.

    So, I added the property schema to my “schemas” project, rebuilt, and sure enough, all the expected promoted values showed up in the orchestration. Now, I’d bet (as in my old example) that the engine can still promote the values with no problem. But, the design time (and maybe the orchestration runtime) has issues with this setup. Either way, seems safe to say that you should keep your property schemas alongside the implementation schemas. This means, ignore the old post and file it under the “Seroter Corollary.” That is, when all else fails, let’s assume I’m an idiot.

    Technorati Tags:

  • Performance Showdown Between BRE and WF

    If you’ve got a couple hours free, and are interested in the performance of the two primary business rules offerings from Microsoft, check out the latest post by Charles Young.

    Charles does some comically thorough analysis and comparison of the Windows Workflow rules engine and the Microsoft Business Rules Engine that ships with BizTalk Server. He looks at performance with relation to rule set size, startup time, fact size, caching and so forth. His conclusion is that the Microsoft Business Rules Engine generally performs better than the WF Rules Engine. However, there are lots of considerations that go into that conclusion, so I heartily encourage you to read and digest Charles’ post.

    Technorati Tags:

  • BizTalk Pattern For Scheduled “Fan Out” Of Database Records

    We recently implemented a BizTalk design pattern where on schedule (or demand), records are retrieved from a database, debatched, returned to the MessageBox, and subscribed to by various systems.

    Normally, “datastore to datastore” synchronization is a job for an ETL tool, but in our case, using our ETL platform (Informatica) wasn’t a good fit for the use case. Specifically, handling web service destinations and exceptions wasn’t robust enough, and we’d have to modify the existing ETL jobs (or create new ones) for each system who wanted the same data. We also wanted the capability for users to make “on demand” request for historical data to be targeted to their system. A message broker made sense for us.

    Here are the steps I followed to create a simple prototype of our solution.

    Step #1. Create trigger message/process. A control message is needed to feed into the Bus and kick off the process that retrieves data from the database. We could do straight database polling via an adapter, but we wanted more control than that. So, I utilized Greg’s great Scheduled Task Adapter which can send a message into BizTalk on a defined interval. We also have a manual channel to receive this trigger message if we wish to run an off-cycle data push.

    Step #2. Create database and database schemas. I’ve got a simple test table with 30 columns of data.

    I then used the Add Generated Items wizard to build a schema for that database table.

    Now, because my goal is to retrieve the dataset from the database, and then debatch it, I need a representation of the *single* record. So, I created a new schema, imported the auto-generated schema, set the root node’s “type” to be of the query response record type, and set the Root Reference property.

    Step #3. Build workflow (first take). For the orchestration component, I decided to start with the “simple” debatching solution, XPath. My orchestration takes in the “trigger” message, queries the database, gets the batched results, loops through and extracts each individual record, transforms the individual record to a canonical schema, and sends the message to the MessageBox using a direct-bound port. Got all that?

    When debatching via XPath, I use the schema I created by importing the auto-generated SQL Server schema.

    Note: If you get an “Inner exception: Received unexpected message type ” does not match expected type ‘http://namespace#node’. Exception type: UnexpectedMessageTypeException” exception, remember that you need an XmlReceive pipeline on the SQL Adapter request response send port. Otherwise, the type of the response message isn’t set, and the message gets lost on the way back to the orchestration.

    Step #4. Test “first take” workflow. After adding 1000 records to the table (remember, 30 columns each), this orchestration took about 1.5 – 2 minutes to debatch the records from the database and send each individual record to the MessageBox. Not terrible on my virtual machine. However, I was fairly confident that a pipeline-based debatching would be much more efficient.

    So, to modify the artifacts above to support pipeline-based debatching, I did the following steps.

    Step #1. Modify schemas. Automatic debatching requires the pipeline to process an envelope schema. So, I took my auto-generated SQL Server schema, set its Envelope property to true, and picked the response node as the body. If everything is set up right, then the result message of the pipeline debatching is that schema we built that imports the auto-generated schema.

    Step #2. Modify SQL send port and orchestration message type. This is a good one. I mentioned above that you need to use the XmlReceive pipeline for the response channel in the SQL Server request-response send port. However, if I pass the response message through an XmlReceive pipeline with the chosen schema set as an “envelope”, the message will debatch BEFORE it reaches the orchestration. Then I get all sorts of type mismatch exceptions. So, what I did, was change the type of the message coming back from the request-response port to XmlDocument and switched the physical send port to to a passthrough pipeline. Using XmlDocument, any message coming back from the SQL Server send port will get routed back to the orchestration, and using the passthrough pipeline, no debatching will occur.

    Step #3. Switch looping to use pipeline debatching. In BizTalk Server 2006, you can call pipelines from orchestrations. I have a variable of type Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMessages, and then (within an Atomic Scope), I called the *default* XmlReceive pipeline using the following code:

    rcvPipeOutputMsgs =
    Microsoft.XLANGs.Pipeline.XLANGPipelineManager
    .ExecuteReceivePipeline(typeof(Microsoft.BizTalk.DefaultPipelines.XMLReceive),
    QueryWorkforce_Response);

    Then, my loop condition is simply rcvPipeOutputMsgs.MoveNext(), and within a Construct shape, I can extract the individual, debatched message with this code:

    //WorkforceSingle_Output is a BizTalk message
    WorkforceSingle_Output = null;
    rcvPipeOutputMsgs.GetCurrent(WorkforceSingle_Output);

    Step #4. Test “final” workflow. Using the same batch size as before (30 columns, 1000 records), it took between 29-36 seconds to debatch and return each individual message to the MessageBox. Compared to nearly 2 minutes for the XPath way, pipeline debatching is significantly more efficient.

    So, using this pattern, we can easily add subscribers to these database-only entities with very little impact. One thing I didn’t show here, but in our case, I also stamp each outbound message (from the orchestration) with the “target system.” The trigger message sent from the Scheduled Task Adapter will have this field empty, but if a particular system wants a historical batch of records, we can now send an off-cycle request, and have those records only go to the Send Port owned by that “target system”. Neat stuff.

    Technorati Tags:

  • Tool: BizTalk Send Port Duplicator

    Often times during development, and even in production, you have a need to create new BizTalk ports that are virtually identical to an existing one.

    For instance, in a “content based routing” scenario, odds are you would test this by creating multiple send ports, all with a slight deviation in subscription criteria and destination path. We also have a case where data received by SAP is sent to a series of virtually identical send ports. Because all the SOAP send ports use the same proxy class for transmitting the service, the ONLY difference is the URL itself. But, it’s a hassle to create a new send port each time.

    So, I took a few minutes yesterday, and using a BizTalk SDK example as inspiration, wrote a small tool that duplicates existing send ports. If I felt more ambitious I’d make it a custom MMC action, but alas, I’m not that motivated.

    When you fire the BizTalk Send Port Duplicator (patent pending) up, the first thing you do is set the server where the BizTalk Management Database resides.

    Then, you optionally choose which BizTalk “application” has the port you wish to copy. If you don’t choose an application from the drop down list, then you’ll get all send ports in your BizTalk environment.

    Next, select the send port from the listbox, and type in a name that will be used for the port copy.

    The copied send port shows up in the BizTalk Administration Console in the same “application” as the source send port. You can open it up and see nearly all properties copied across. What properties are included? You can copy one-way or two-way ports, filter/subscription, primary transport details (address, retry, handlers), maps, and pipelines. At the moment, I’m not copying secondary transport details, certificates, tracking details, or dynamic ports.

    You can download the BizTalk Send Port Duplicator here. I’ve included the source code as well, so feel free to mess around with it.

    [UPDATE (10/02/2007)] I’ve updated the application to also support choosing the name of the Management database. Prior, I had hard-coded this value.

    Technorati Tags:

  • CTP3 of ESB Guidance Released

    Some very cool updates in the just-released CTP3 of ESB Guidance. The changes that caught my eye include:

    • Download the full Help file in CHM format. Check out what’s new in this release, sample projects, and a fair explanation of how to perform basic tasks using the package.
    • New endpoint “resolver” framework. Dynamically determine endpoint and mapping settings for inbound messages. Interesting capability that I don’t have much use for (yet).
    • Partial support for request/response on-ramps. An on-ramp is the way to generically accept messages onto the bus by receiving in an XmlDocument parameter. I’ll have to dig in and see what “partial support” means. Obviously the bus would need to send a response back to the caller, so I’ll be interested to see how that’s done.
    • BizTalk runtime query services. Looks like it uses the BizTalk WMI interfaces to pull back information about hosts, applications, messages, message bodies and more. I could see a variety of ways I can use this to surface up environment data.
    • SOA Software integration. This one excites me the most. I’m a fan (and user) of SOA Software’s web service management platform, and from the looks of it, I can now more easily plug in any (?) receive location and send port into Service Manager’s monitoring infrastructure. Nice.

    I also noticed a few things on Exception Management that I hadn’t seen yet. It’s going to be a pain to rebuild all my existing ESB Guidance Exception Management solution bits, so I’ll wait to recommend an upgrade until after the final release (which isn’t far off!).

    All in all, this is maturing quite nicely. Well done guys.

    Technorati Tags: ,