I just finished zipping up all the source code for my recent set of articles over at TopXML.com. Specifically, I just added the source code for the set of articles on publishing WCF services out of BizTalk (with security, transactions, attachments) and the source code for all the BizTalk Adapter Pack demonstrations that utilized the Oracle adapter. I make no promises that the code is attractive, contains best practices, or avoids the use of obscenities in the comments.
One of our BizTalk developers had a requirement to collect related messages and send a single email summary with details about those messages. I built the example below to help her out.
In this scenario, a series of independent, but related, messages are sent to BizTalk. Each of these messages will have a “batch ID” which connects them as well as a “batch count” value which identifies the total number of messages in the batch. As you might expect, I’ll need a convoy to collect these messages. The interesting part was how to build up the email data which summarized the results of the processing for a given batch.
First, I have a schema that represents an individual message.
As you can see, I have details about the batch, and then a record containing details about the document that was processed through BizTalk. These fields are mostly populated by the system called by BizTalk earlier in the process and those new values need to be reported back to the initiator of the submission.
The next schema represents the email content being sent back to the initiator. There is a summarization of the batch of records they submitted, and then one section where successfully processed documents are recorded, and a section where failed documents are recorded.
Next, we need a convoy orchestration that processes all messages for a given batch. The first receive shape initializes a correlation set on “batch ID” (from a separate property schema) and then initializes the loop variables. This loop will run until all the number of messages received is equal to the “batch count” in the message.
The meat of this orchestration is the part that builds up the email message. I have a helper class that accepts data from each batch message, and stores it in a member variable until I’m ready to return the completed, aggregate message. Now, I could have chosen to build some sort of custom type object, and when the loop was complete, turn that object into the XML representation of my email schema. But, I’d rather cut out that middle man. So, I passed my “BatchSummary” schema (above) through the .NET Framework xsd.exe tool to get a type object that directly mapped to the schema. That type is used as a member variable of my helper class.
You’ll notice that I also created a few lists to hold the success and failure item types. The “BatchSummary” object takes in an array of success and failure items, but during the running of the convoy, I have no idea how many success or failure items I have, and thus couldn’t properly initialize an array of the necessary size. So, by creating a list, and simply adding to it along the way, I can postpone array creation until later.
Within this class I have operations to add message data elements to the appropriate “success” or “failure” list object, and then finally, the convoy orchestration should call the operation below to get a “completed” batch summary object.
The next part is fun. I created an orchestration message for the batch email message, but, instead of choosing the XSD file for the “Message Type”, I chose the object generated by the xsd.exe tool.
This .NET object has all the necessary metadata to automagically serialize into an XML message on the way out of the orchestration. At the end of the orchestration loop, I have a “message assignment” shape where I create the orchestration message by calling the appropriate operation on my helper class.
Because I want to take this XML payload and turn it into an HTML email, I need to massage the data on the way out. For this scenario, I used the XslTranformComponent sample from the BizTalk SDK (C:\Program Files\ Microsoft BizTalk Server 2006\ SDK\ Samples\ Pipelines\ XslTransformComponent). After building this pipeline component and GAC-ing it, I created a new send pipeline and dropped this component there. Finally, I wrote an XSLT stylesheet which took the XML and prettied it up. Now, when I drop three files (all with the same “batch ID”) into BizTalk, I get the following email:
Nice! So, with fairly few moving parts, I collected a bunch of related messages, built up a new composite message on the fly, and then send a .NET object-type orchestration message out, which had an XSLT transform applied before being emailed to the target recipient.
Technorati Tags: BizTalk
The latest issue of the BizTalk HotRod magazine is out. Some of the topics you’ll find are:
And much more. As usual, a well done issue.
Skimming through this issue got me thinking about where I get my BizTalk information, and reminded me to update my RSS reader so that I regularly read some of the “newer guys” who cover BizTalk topics. Some of the original BizTalk giants (like Steven, Tomas, Charles, Scott, Jon, Lee, etc) have (naturally) shifted some of their attention to other technologies, so it’s important to keep an eye out for folks who are taking a fresh look at BizTalk things.
Some of the blogs I read on occasion (but need to actually subscribe to) include:
I need a show of hands of who still has Scott Woodgate on their BizTalk blogroll. Seriously people, it’s time to let go.
Technorati Tags: BizTalk
I’m currently architecting a project where one of the requirements is to limit the number of concurrent calls to a web service. I’d covered a similar topic in a previous post, and outlined two ways one could try and configure this behavior.
First, you could limit the number of simultaneous connections for the SOAP adapter by setting the “maxconnections” setting in the btsntsvc.exe.config file. The downside to this mechanism is that if you have many messages, and the service takes a while to process, you could timeouts.
The second choice is to turn on ordered delivery at the send port. This eliminates the timeout issue, but, really slows processing. In our case, the downstream web service (a FirstDoc web service that uploads documents to Documentum) is fairly CPU intensive, and may take upwards of 200 seconds to run (which is why I also asked the developer to consider an asynchronous callback pattern), so we need a few calls happening at once, but not so many that the box collapses.
So, Scott Colestock recommended I take a look at the last issue of the BizTalk HotRod magazine and review the orchestration throttling pattern that he had expalined there. Unlike the two options mentioned above, this DOES requirement development, but, it also provides fairly tight control over the number of concurrent orchestrations. Since Scott’s article didn’t have code attached, I figured I’d rebuild the project to help our developer out, and, learn something myself.
The first step was to define my two BizTalk message schemas. My first is the schema that holds the data used by the FirstDoc web service. It contains a file path which the web service uses to stream the document from disk into Documentum. I didn’t want BizTalk to actually be routing 50MB documents, just the metadata. The second schema is a simple acknowledgement schema that will be used to complete an individual processing instance. Also, I need a property schema that holds a unique correlation ID, and the instance ID of the target convoy. Both properties are set to MessageContextPropertyBase since the values themselves don’t come from the message payload but rather, the context.
The second step was to build a helper component which will dictate which throttled instance will be called. Basically, this pattern uses convoy orchestrations. The key is though, that you have multiple convoys running, vs. a true singleton that processes ALL messages. The “correlation” used for each convoy is an instance ID that corresponds to a number in the numerical range of running orchestrations allowed. For instance, if I allowed 10 orchestrations to run at once, I’d have 10 convoy orchestrations, each one initializing (and following) an “InstanceID” between 1-10. Each of my calling orchestrations acquire a number between 1-10, and then target that particular correlation. Make sense? So I may have 500 messages come in at once, and 500 base orchestrations spin up, but each one of those target a specific throttled (convoy) orchestration.
So my helper component is responsible for doling out an instance ID to the caller. The code looks like this:
Notice that because it’s a static class, and it has a member variable, we have to be very careful to build this in a thread safe manner. This will be called by many orchestrations running on many threads.
Once this component was built and GAC-ed, I could build my orchestration. The first orchestration is the convoy. The first receive shape (which is direct bound to the MessageBox) initializes the “instance ID” correlation set. Then I have a loop which will run continuously. Inside that loop, I have a placeholder for the logic that actually calls Documentum, and waits for the response. Next I build the “acknowledgement message”, making sure to set the “Correlation ID” context property so that this acknowledgement reaches the orchestration that called it. I then send that message back out through a direct bound send port. Finally, I have a receive shape which follows the “Instance ID” correlation set (thus defining this orchestration as a convoy).
Next, we have the orchestration that spins up for EVERY inbound message. First, it receives a message from a port bound to a file receive location. Next, within an Expression shape, I call out to my “round robin” helper component which gives me the next instance ID in the sequence.
I then make sure to create a new message with both the “Correlation ID” and “Instance ID” context properties set.
Finally, I send this message out (via direct bound send port) and wait for the acknowledgement back.
So what I have now is a very (basic) load balancing solution where many inbound messages flow through a narrowed pipe to the destination. The round robin helper component keeps things relatively evenly split between the convoy orchestrations, and I’m not stuck using a singleton that grinds all parallel processing to a halt. Running a few messages through this solution yields the following trace …
If I look in the BizTalk Administration Console, I now have three orchestrations running at all times, since I set up a maximum of three convoys. Neat. Thanks to Scott for identifying this pattern.
Any other patterns for this sort of thing that people like?
Technorati Tags: BizTalk
So the documentation that comes with the BizTalk Adapter Pack makes scant reference to flowing transactions to the adapters. That is, if I want to call the “Insert” operation on an “Orders” table, but only commit that if the “Insert” operation on the “Order Items” table succeeds, how do I wrap those operations in a single transaction?
WCF has great transaction support, and the BizTalk Adapter Pack is built on WCF, but the product documentation for the Oracle adapter states:
The Oracle Database adapter does not support performing transactions on the Oracle database using System.Transaction. The adapter supports transactions using OracleTransaction.
Hmmm. That’s pretty much the only time transactions are mentioned at all. That makes it sound like I cannot wrap my service calls in a System.Transaction and have to use the OracleTransaction object from the ODP.NET bits. What better way to confirm this than by actually testing it?
I’m using the example from my TopXML.com articles. So in that article, I mention inserting into two tables sequentially via proxy classes. So, what happens if I take that same block of “insert” code and purposely create an error in the second set of data (e.g. use a non-existent “OrderID”)? An exception occurred during the second operation, but the first insert command succeeded …
Notice that my “Orders” table has a record in it, but the “OrderItems” table has no corresponding items for OrderID #34. So, I’m stuck in an inconsistent state. Not good.
On a whim, I decided to wrap the entire block of “insert” code inside a System.Transaction.TransactionScope block to see what would happen. On the first execution, I got an error saying “Unable to Load OraMTS“. Interesting. Looked like the System.Transaction in my code is converted to an Oracle transaction by the adapter and the OraMTS object (from the Oracle client) wasn’t found. So, I went back to my Oracle client installation and made sure to install the Oracle Services for Microsoft Transaction Server.
Now, if I executed my code again, with the same error in the 2nd set of insert commands, the database remained in a consistent state, and the first insert did not commit. So you CAN wrap these service invocations inside a System.Transacton object (at least for the Oracle adapter) to daisy-chain atomic operations.
Overall, the documentation for the BizTalk Adapter Pack is top notch, but the complete absence of transaction instructions seems curious.
Earlier this week, I grabbed a couple new CDs, hopped in the car, and drove down to San Diego to present at the .NET User Group’s Connected Systems meeting. The topic was the BizTalk Adapter Pack and I outlined what the BAP is, what the core use cases are, and demonstrated how to build a solution using it.
My presentation is here. I called out a few resources for folks looking to learn about the BizTalk Adapter Pack. They include:
I’ve been accepted into the Microsoft TAP for the Adapter Pack Office Developers Program, so hopefully I’ll be able to demonstrate how to use the BAP within Office applications.
Michele recently wrote a lengthy but deliciously thorough article on MSDN (hat tip: Lynn) called Application Deployment Strategies which identifies five core scenarios for using Windows Communication Foundation within a distributed application. Those scenarios include:
For each scenario, Michele nicely calls out the implementation characteristics (e.g. hosting environment, authentication, etc), configuration details, and a set of common questions and answers (e.g. “What other credential types are supported for web services?” or “When should I use a reliable session?”).
The formatting of the article itself makes it a bit tricky to understand which header topic you’re currently under, but, I’m sure you’re smart enough to figure it out.
UPDATE: I have since moved these articles to my own blog and they can be found here.
Whew. My 9th and final article for TopXML.com on integrating BizTalk Server and WCF is now published. This one looks at the BizTalk Adapter Pack and how to consume the Oracle adapter from inside BizTalk Server. I demonstrated how to insert multiple records at once, call stored procedures, build a database polling solution, and how to call the Adapter Pack from an orchestration using code.
20,500+ words and 178 screenshots later, I definitely learned a lot about WCF (and Oracle) by writing this series of articles. I found the security and attachment topics to be challenging, the transaction topics to be quite interesting, and the BizTalk Adapter Pack to be quite compelling. The WCF integration in BizTalk Server 2006 R2, while not as rich as it could be, is actually pretty darn well built and thought out.
When I started out on this series, I figured that writing about BizTalk/WCF and then the BizTalk Adapter Pack would be somewhat “niche” for the majority of developers, but, hopefully this helps people down the line as they begin to really investigate and evaluate these technologies together.
UPDATE: I have since moved these articles to my own blog and they can be found here.
So how would you use the BizTalk Adapter Pack to build a RESTful HTTP URI on top of an Oracle database table? What about calling Oracle stored procedures that made use of either strong or weak ref cursors? In my latest article for TopXML.com, I explore how to consume the Microsoft BizTalk Adapter Pack’s Oracle adapter using the WCF service model.
Specifically, I show how to …
This was a fun article to write. The ability of the adapter to poll an Oracle database and kick off an event introduces a number of possibilities. Also, applying some of the things that I had read about the new RESTful WCF attributes with the auto-generated Oracle adapter bits proved to me a useful exercise.
Feedback? Let me know.
Today, Steve Martin announced a few details about the next version of BizTalk that will be available in early 2009. Called BizTalk Server 2006 R3, this amounts to an incremental release of the product that does NOT correspond to the Oslo wave. The “headline” for this release is compatibility with the latest version of the Microsoft platform (Windows Server 2008, SQL Server 2008, Visual Studio.NET 2008). Looks like a couple additional nuggets will be added, such as new and enhanced adapters (such as WCF SQL adapter?).
I’m not a fan of the naming. Why not call it BizTalk Server 2000 R7? If the software is going to have a “year” attached to its name, seems like a good idea to actually increment it. All joking aside, the name makes SOME sense because this is really just a service pack on the existing BizTalk Server 2006 code base. Calling it BizTalk Server 2009 would create a perception of a brand new product, which this is not.
Technorati Tags: BizTalk
Richard Seroter's Architecture Musings 