One of the technology backbones of my company moving forward is the use of BizTalk to “fan out” messages from our ERP system (SAP). Instead of having hundreds of direct interfaces to SAP by downstream systems who need real-time data, we can use BizTalk to receive a single message from SAP (XI), and distribute it to all interested parties. One issue brought up recently was “ordered delivery” and ensuring that if a downstream system needs messages delivered in the order sent by SAP, that BizTalk honor it. So, I set out to test a variety of scenarios using BizTalk’s built-in ordered delivery capabilities.
Setup: I exposed a BizTalk schema as a web service to simulate our publishing receiver from SAP. A simple Windows Form is sending messages to this web service, and stamping each one with a sequence number. I then built a couple of web services to act as “subscribers” of these SAP data events. These web services publish the data to a database table, thus letting me see the exact order of messages delivered. Each SOAP send port has a subscription based on message type.
Scenario #1: Send port, no ordered delivery
When a series of messages are published to the receiving web service and routed to the subscriber without ordered delivery, the result may look like this:
As you can see, even on my single development machine, the order gets mixed up. This is due to batch processing and the multi-threaded nature of BizTalk message distribution.
Scenario #2: Send port, ordered delivery turned on
To turn ordered delivery on, all we need to do is check a box on the send port (this assumes that your inbound transport can deliver messages in order. Examples include MSMQ, SOAP and HTTP).
The result in the database looks like this:
So you can see that all the messages are in sequential order.
Scenario #3: One send port no ordered delivery, one send port WITH ordered delivery
I wanted to prove that ordered delivery only impacted the corresponding port. When sending the message into BizTalk with two send ports subscribing, the data looked like this:
“Subscriber #1” kept everything in order, but “Subscriber #2” delivered messages a bit more haphazardly.
Scenario #4: Non-ordered delivery send port, no retries, and error in service
So what if the service raises an error? When there are no retries, and the send port doesn’t have an ordered delivery requirement, the result looks like this:
The bad message (#12) is simply skipped (because it is now suspended).
Scenario #5: Ordered delivery send port, retries enabled, and error in service
How about for an ordered delivery send port? What if there’s an error in the service call, and retries are turned on for the port? Also, the “cancel if error” is turned off. Check this out:
The messages queue up waiting for the first one to retry, and hopefully succeed. We’ll see more of this in a moment.
Scenario #6: Ordered delivery send port, NO retries, and error in service
This time, there’s an error in the service, and no retries. Also, the “cancel if error” flag is still turned off. The result is:
So the messages still get delivered AFTER the bad message has been encountered. That’s expected, since we told BizTalk to keep going, even for the ordered delivery port.
Scenario #7: Ordered delivery send port, NO retries, “cancel if error” turned on, and error in service
What if we reproduce the previous scenario, BUT, turn “cancel if error” on? This flag can be set on the send port here:
What you’re telling BizTalk is that if any message fails for this ordered delivery port, stop processing until this error can be corrected. This is useful if you’re concerned that an “insert new contact” message failed, but you expect a “modify existing customer” to be following. Clearly the “modify” message will fail unless the “insert” gets figured out. The result of this?
Processing stopped after message #11 because message #12 failed. The send port is still “started”, but you’ll find a suspended message. If you open it up, you’ll see that all following messages are queued up until the offender gets resolved.
Scenario #8: Send port stopped, and restarted for a non-ordered delivery send port
What if we KNOW that a downstream system is unavailable (e.g. maintenance) and want to prevent the inevitable failure of delivery? Maybe we want to shut off the send port, queue up the messages, and once the unavailable system is back online, open the distribution pipeline again. For a non-ordered delivery port, after restarting a stopped send port, the output looked like this:
The messages are in a crazy order, as you can see.
Scenario #9: Send port stopped, and restarted for an ordered delivery send port
Final scenario. What if we stop an ordered delivery port, and then start it back up later? Do we retain the correct ordering? When I sent five messages in, I got a suspended instance with five messages:
If I send in three MORE messages, they actually get added to this suspended instance:
Once I finally turn the subscriber service back on, all messages were sent in the same order received.
So there you go. Ordered delivery is a fairly powerful concept, and not particularly hard to do with BizTalk. It greatly impacts performance because the send port won’t send subsequent messages until a delivery confirmation is received, but, in many cases that performance impact is outweighed by business requirements.
Technorati Tags: BizTalk
Richard Seroter's Architecture Musings 
Hi Richard,
Clear overview, nice to see the results of all the scenario’s you tested. There is 1 functionality you might want to highlight too and add to your post. For ‘Scenario #7: Ordered delivery send port, NO retries, “cancel if error” turned on, and error in service’. If the service is stopped you can right-click and do a ‘find failed message’. Subsequently you can have a look at the msg content, you can kill the msg that triggered the error in case it is a ‘poison msg’ and just resume the stopped service if you want, all inside the admin console.
Cheers,
Grego
Is that right? Sheesh, I have to try that. That makes perfect sense, since if the “failed message” is inherently invalid, it will NEVER work, and thus block things up indefinitely.
Great comment, thanks.
Great post, Richard. Quick question — are you saying that the sequence number is a configurable setting? In other words, is there a way I can determine what the sequence number will be in *any* message type coming in? How can I go about configuring/controlling something like that?
Thanks!
Hey buddy,
I actually injected the “sequence number” as a node in the message itself. For SOAP messages, you could possibly make this a SOAP header element, thus never modifying the message payload itself.
Very nice documentation of your tests.
I’m testing Ordered Delivery for IDocs generated by SAP. My scenario involves BizTalk being down. (never happens, right 🙂 If, hold, when the BTS server goes down and SAP generates an IDoc the message is put in a retry job queue. Being a BizTalk Ninja I quickly restore BizTalk. The next IDoc generated is successful. Once the retry interval passes the failed message is sent. Ordered delivery is now broken.
Have you experienced this in your environment? Do you have any advice on designing for this?
Hey AdemusPrime,
We’ve got a similar situation with SAP. By default, BizTalk respects ordered delivery once it gets the messages. Clearly in this case, BizTalk will still process them in order, BUT technically it’s now out of order.
You could try a re-sequencer pattern and try to “wait” for queued messages. In our case, we’ll probably ensure that each downstream subscriber can handle BizTalk based retries (so if an “update” beats an “insert”, the “update” will try again in 5 minutes after we can be sure the “insert” made it), or gracefully handle the messages if they fall out of sequence.
Great post Richard; thanks. What about the scenario where there’s one or more orchestrations in the mix? In other words, one or more orchestrations must process the message prior to sending. In such a scenario you would encounter out-of-order messages due to the variations in throughput timing of the orchestration(s) based on message size and workflow activities. We had a scenario with our healthcare solution that required ordered delivery of HL7 messages which required transformation and orchestration processing. BizTalk 2006’s order delivery send port capability will not solve the problem. What I’d like to see in the next version of BizTalk is a core capability to preserve message order for the original message and all “children” of the original message (i.e. new message instances derived from the original) regardless of the activities that take place in BizTalk. This could be achieved if each message received by BizTalk were immediately “stamped” with a sequence number that would automatically be inherited by all “children” messages, and then the subscription “engine” would pick up message in sequence number order. This approach should allow the intervening workflows/orchestrations to be executed by BizTalk without restricting the execution to be sequential (i.e. ordered delivery remains the responsibility of the send ports).
Any thoughts on this? I’ve been “away” from BizTalk for a while and may have some of my facts wrong, so please correct away.
Hi Steve,
Good comment. If you are truly chaining the orchestrations, you could potentially make EACH a singleton (required any time you do ordered delivery where orcheestration is involved). Messages get processed by the first orchestration, picked up (still in order) by the second, and finally dropped back for the send port in the original order. I’d have to see if that works.
Agreed though, that a “stamping” function can be useful, especially in ordered delivery (or sorting) scenarios.
Through as always – excellent post.
@ Steve Hart: Here’s a pattern Microsoft published a while ago (http://msdn.microsoft.com/en-us/library/bb851740.aspx) when it comes to keeping the order when multiple orchestration are in play. Don’t know if you seen it but it might be of interest.
I’ll also like to recommend this webcast on the subject in general (http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?EventID=1032288276&Culture=en-US). I love the “drawing” demo!
Richard Hallgren
Hi Richard,
Is there anyway i can force a nonresumable message through a send port which has ordered delivery enabled?
I’m not sure you can force a non-resumable message through anything at all. Probably need another way to extract and resubmit the message, and just accept it will be out of order. Unless any other readers here have an alternate idea …
Very informative piece.
Richard, brilliant article, as usual.
I have a singleton sequential convoy orchestration (4 of them actually) and am considering replacing each of them with a send port with a custom assembler pipeline component doing the work with ordered delivery on the port. would this be a bad idea? I dont see that the orchestration does any work that could not be achieved in the custom pipeline component …
How does this sound?. I know I’ve given no detail on my implementation but it is a classic uniform sequential convoy. the OD functionality of the Send port will geve me the singleton functionality that I got from the convoy orchestration, the custom pipeline component would do the batching for me.
Something deep down tells me this could improve performance, removing orchestration from the picture and replacing it with messaging. Again I am not necessarilly interested in the ordered delivery, but what I get out of the box from BizTalk by turning OD on on the send port coupled with my custom assembler component, i.e. a convoy in a send port!!
Would I be mad to consider such a solution?
Hi Liam. Interesting idea. What would you be batching? What is your existing convoy doing? Assuming you want to try and maintain state in a pipeline, it’s potentially risky if anything fails, you lose it.
Good article. Does anyone know how to control SOAP send port number of simultaneous calls to the web service. For making more than one call ordered delivery has to be unchecked but that opens the send port for as many calls as set in the machine config. If I change machine config then it will limit calls to all web services made from that machine. I am trying to limit calls to one particular web service.
What if my sendport is WCF-SQL, and SQL goes down for 30 minutes in the middle of the night (and we are processing 20,000 messages an hour). Would the best scenario be your option 7, but with retries turned on? Then no human intervention required, and the messages are still in order, correct?
Neal, I think that’s right; assuming that you have either (a ) enough retries, or (b ) long enough intervals between them. Otherwise you’ll come back into work with a 100k backlog!
I was also thinking that this would impact your policy/procedure on how to handle/catch errors in Stored Procs. If an occasional message caused a stored proc error, then it hold up everyone else “in-line”. Thus the errors would have to be trapped, logged to some error table, and reported by other means.