Author: Richard Seroter

Q: Congrats on new version of BizTalk Server.  At my company, we just deployed BTS 2006 R2 into production.  I’m sure many other BizTalk customers are fairly satisfied with their existing 2006 installation.  Give me two good reasons that I should consider upgrading from BizTalk 2006 (R2) to BizTalk 2009.

A: Thank you Richard for the opportunity to answer your questions, which I’m sure are relevant for many existing BizTalk customers.

I’ll be more generous with you (J) and I’ll give you three reasons why you may want to upgrade to BizTalk Server 2009: to reduce costs, to improve productivity and to promote agile innovation. Let me elaborate on these reasons that are very more important in the current economic climate:

1. Reduce Costs – through servers virtualization and consolidation and integration with existing systems. BizTalk Server 2009 supports Windows Server 2008 with Hyper-v and SQL Server 2008. Customers can completely virtualize their development, test and even production environments. Using less physical servers to host BizTalk solutions can reduce costs associated with purchasing and maintaining the hardware. With BizTalk Enterprise Edition you can also dramatically save on the software cost by running an unlimited number of virtual machines with BizTalk instances on a single licensed physical server. With new and enhanced adapters, BizTalk Server 2009 lets you re-use existing applications and minimize the costs involved in modernizing and leveraging existing legacy code. This BizTalk release provides new adapters for Oracle eBusiness Suite and for SQL Server and includes enhancements especially in the Line of Business (LOB) adapters and in connectivity to IBM’s mainframe and midrange systems.
2. Improve Productivity – for developers and IT professionals using Visual Studio 2008 and Visual Studio Team System 2008 that are now supported by BizTalk. For developers, being able to use Visual Studio version 2008 means that they can be more productive while developing BizTalk solutions. They can leverage new map debugging and unit testing options but even more importantly they can experience a truly connected application life cycle experience. Collaborating with testers, project managers and IT Pros through Visual Studio Team System 2008 and Team Foundation Server (TFS) and leveraging capabilities such as: source control, bug tracking, automated testing , continuous integration and automated build (with MSBuild) can make the process of developing BizTalk solutions much more efficient. Project managers can also gain better visibility to code completion and test converge with MS project integration and project reporting features. Enhancements in BizTalk B2B (specifically EDI and AS2) capabilities allow for faster customization for specific B2B solutions requirements.
3. Promote Agile Innovation – specific improvements in service oriented capabilities, RFID and BAM capabilities will help you drive innovation for the business. BizTalk Server 2009 includes UDDI Services v3 that can be used to provide agility to your service oriented solution with run-time resolution of service endpoint URI and configuration. ESB Guidance v2 based on BizTalk Server 2009 will help make your solutions more loosely coupled and easier to modify and adjust over time to cope with changing business needs. BizTalk RFID in this release, features support for Windows Mobile and Windows CE and for emerging RFID standards. Including RFID mobility scenarios for asset tracking or for doing retail inventories for example, will make your business more competitive. Business Activity Monitoring (BAM) in BizTalk Server 2009 have been enhanced to support the latest format of Analysis Services UDM cubes and the latest Office BI tools. These enhancements will help decision makers in your organization gain better visibility to operational metrics and to business KPI in real-time. User-friendly SharePoint solutions that visualize BAM data will help monitor your business execution ensure its performance.

Q: Walk us through the process of identifying new product features.  Do such features come from (a) direct customer requests, (b) comparisons against competition and realizing that you need a particular feature to keep up with others, (c) product team suggestions of features they think are interesting, (d) somewhere else  or some combination of all of these?.

A: It really is a combination of all of the above. We do emphasize customer feedback and embrace an approach that captures experience gained from engagements with our customers to make sure we address their needs. At the same time we take a wider and more forward looking view to make sure we can meet the challenges that our customers will face in the near term future (a few year ahead). As you personally know, we try to involve MVPs from the BizTalk customer and partner community to make sure our plans resonate with them. We have various other programs that let us get such feedback from customers as well as internal and external advisors at different stages of the planning process. Trying to weave together all of these inputs is a fine balancing act which makes product planning both very interesting and challenging…

Q: Microsoft has the (sometimes deserved) reputation for sitting on the sidelines of a particular software solution until the buzz, resulting products and the overall market have hit a particular maturation point.  We saw aspects of this with BizTalk Server as the terms SOA, BPM and ESB were attached to it well after the establishment of those concepts in the industry.  That said, what are the technologies, trends or buzz-worthy ideas that you keep an eye on and influence your thinking about future versions of BizTalk Server?

A: Unlike many of our competitors that try to align with the market hype by frequently acquiring technologies and thus burdening their customers with the challenge of integrating technologies that were never even meant to work together, we tend to take a take a different approach. We make sure that our application platform is well integrated and includes the right foundation to ease and commoditize software development and reduce complexities. Obviously it take more time to build an such an integrated platform based on rationalized capabilities as services rather than patch it together with foreign technologies. When you consider the fact that Microsoft has spearheaded service orientation with WS-* standards adoption as well as with very significant investments in WCF – you realize that such commitment have a large and long lasting impact on the way you build and deliver software.
With regard to BizTalk you can expect to see future versions that provide more ESB enhancements and better support for S+S solutions. We are going to showcase some of these capabilities even with BizTalk Server 2009 in the coming conferences.

Q: We often hear from enlightened Connected Systems folks that the WF/WCF/Dublin/Oslo collection of tools is complimentary to BizTalk and not in direct competition.  Prove it to us!  Give me a practical example of where BizTalk would work alongside those previously mentioned technologies to form a useful software solution.

A: Indeed BizTalk does already work alongside some of these technologies to deliver better value for customers. Take for example WCF that was integrated with BizTalk in the 2006 R2 release: the WCF adapter that contains 7 flavors of bindings can be used to expose BizTalk solutions as WS-* compliant web services and also to interface with LOB applications using adapters in the BizTalk Adapter Pack (which are based on the WCF LOB adapter SDK).

With enhanced integration between WF and WCF in .NET 3.5 you can experience more synergies with BizTalk Server 2009. You should soon see a new demo from Microsoft that highlights such WF and BizTalk integration. This demo, which we will unveil within a few weeks at TechEd North America, features a human workflow solution hosted in SharePoint implemented with WF (.NET 3.5) that invokes a system workflow solution implemented with BizTalk Server 2009 though the BizTalk WCF adapter.

When the “Dublin” and “Oslo” technologies will be released, you can expect to see practical examples of BizTalk solutions that leverage these. We already see some partners, MVPs and Microsoft experts that are experimenting with harnessing Oslo modeling capabilities for BizTalk solution (good examples are Yossi Dahan’s Oslo based solution for deploying BizTalk applications and Dana Kaufman’s A BizTalk DSL using “Oslo”). Future releases of BizTalk will provide better out-of the-box alignment with innovations in the Microsoft Application Platform technologies.

Q [stupid question]: You wear red glasses which give you a distinctive look.  That’s an example of a good distinction.  There are naturally BAD distinctions someone could have as well (e.g. “That guy always smells like kielbasa.”, “That guy never stops humming ‘Rock Me Amadeus’ from Falco.”, or “That guy wears pants so tight that I can see his heartbeat.”).  Give me a distinction you would NOT want attached to yourself.

A: I’m sorry to disappoint you Richard but my red-rimmed glasses have broken down – you will have to get accustomed to seeing me in a brand new frame of a different color… J
A distinction I would NOT want to attach myself to would be “that unapproachable guy from Redmond who is unresponsive to my email”. Even as my workload increases I want to make sure I can still interact in a very informal manner with anybody on both professional and non-professional topics…

Q: Whether someone has just purchased BizTalk Server, or is migrating an existing environment, appropriate solution sizing is critical.   What are the key questions you ask customers (or consider yourself) when determining how best to right size a BizTalk farm (OS, hardware, database, etc)?

A: I’ll start with the numbers that I use and then go through the thought process I use to scope the size of environment I need when I run performance labs.  The number of messages that you can process on a BizTalk Group depends on a lot of factors (machine characteristics, adapters, etc.).  Therefore the best way to size a BizTalk solution remains to do a POC. However if this is not possible, here are the numbers that I can and do use to make sure I am in the right ballpark.  These numbers are derived from our internal test results. The BizTalk machines were Dual Proc Dual Core, 4 GB memory and SQL was Dual Proc Quad Core with 8 GB of memory:

Single Server Messaging Performance

1. A simple small messaging  scenario can achieve 715 messages a second (WSHTTP WCF One Way Messaging Scenario using 2KB messages and passthru pipelines)
2. Optimizing the transport takes it to ~850 messages a second. Utilizing NetTCP provided us with an approx  ~18% gain.

The scale factor that I use is approximately 1.5 per BizTalk Server. So in this scenario going to 2 servers I would expect ~1000-1200 messages per second. At a certain point adding additional BizTalk Servers is going to cause the MessageBox SQL Server to become a bottleneck. To alleviate this Multiple MessageBox’s can then be added. In practice going past 4 or 5 MessageBox’s the returns begin to diminish.

Stage of the Project

The first thing I need to determine is the stage of the project.  If I am coming in at the beginning of the project, I will always seek to understand the customers business problem first, as it is only once I know this that I can ascertain their requirements. Once I understand a customer’s requirements the first question, I ask myself is “is BizTalk the right solution for this customers problem”.  BizTalk is a fantastic product but is not a universal panacea for all problems.  I strongly believe that positioning the right technology to solve the problem a customer has is my number one job, even if this means not using BizTalk.  A common question I am hearing (along with every other BizTalk person) is “when should I use BizTalk over Dublin and vice versa”.  Now answering that would be a full article in itself so rather than attempt it here I will refer to Charles Young’s very good article on the topic here.  I also use the numbers mentioned above to determine whether there system requirements are realistic.

Assuming that BizTalk is the right solution and will fulfill their requirements I then need to understand the key characteristics of the system, specifically I’m interested in the following:

1.  Message flow through the system. 

Now I know that there are often many tens of message flows through a system. For the majority of systems I’ve worked,  a much smaller subset of these tends to account for a large proportion of the load and hence have the most impact of perceived performance of the system.  I’ve found that identifying these and focusing on them is key.  For example, I recently worked with an customer in Europe to test their BizTalk system which was handling the back-end processing for their new Internet Bank.  In this case, over 80% of all requests were for the “Summary of Accounts view” (providing a consolidated view of all bank accounts).  In this scenario users are unlikely to wait a long time when they first log into the online bank therefore optimizing this and any directly message flows should be the key priority.    Once I’ve identified these flows, I’m primarily interested in the features of BizTalk that are being used.  Is messaging used heavily? Orchestration? Business Rules Engine? Is tracking required, either out of the box or BAM? I’m also trying to understand what external system are involved and how many calls are made from Orchestration.  Each of these features has some performance “cost” associated with it, my job as a Ranger is to work with the customer to ensure they get the functionality that they need at the minimum possible cost in terms of performance . In most cases I will put together a Visio diagram of the main message flows and clarify that these are correct with all the Developers of the system.

2.  Size/Format/Distribution of the messages

Message size and format is very important regarding BizTalk performance.  Processing binary files is expensive in terms of performance because often the message body cannot be compressed in the same way that XML can.   It is important to determine how big the messages will be and their distribution.  This is particularly important for large batches of messages that in many systems occur once a week.  I’ve seen many customers present me with extremely complex spreadsheets illustrating each and every single message type that will be processed in the system.  I think it is important to abstract to the appropriate level of detail otherwise dealing with and reproducing this in a lab becomes an impossible task. I typically ask customers to put together a table as per below with the constraint that it must be simple enough to fit on a single PowerPoint slide.

3.  Production Traffic Can This be Replayed

Ideally in any performance testing or scoping scenario, I want to be able to replay actual data that will be processed in production.  Especially when the Rules Engine is used this is important to validate correct functionality.  Testing with a small subset of test data is better than nothing, but it is my experience that using production data will identify issues in system design before they get to production which is good for everybody.

4.  Performance Goals

Clear quantifiable goals are a must for anyone serious about BizTalk! Without these there is no quantifiable way to  judge the effectiveness of the system.  In short, performance goals are essential part of project success criteria. Good goals should state any constraints and should cover throughput and latency and any other relevant factors as well as how you will measure them. I’ve included an example below:

     Orchestration scenario

         Throughput :250,000 calls within 8 hours

                            ~9 messages/sec sustainable

          Latency:    < 3 seconds for  99% of all response messages

5.  Type of Environment

How many applications are present on the system – is this BizTalk group going to be processing a single application or is it going to be a centralized environment?

Q: You’ve done a great job capturing BizTalk performance tuning metrics and providing benchmarks for the results.   The sheer volume of options can seem intimidating, so give me only four modifications you would want all BizTalk owners to make in their environment.

A: I decided to cheat a bit on this answer and break my four modifications into areas: hardware, sql config, BizTalk config and monitoring.

1.  Invest in the right hardware – gigabit networking, fast storage (SAN or 15K local SQL disks), modern fast processing machines.

2.  Optimize SQL Server configuration for BizTalk. Including:

• Data and log file separation
• Create separate Temp DB data files (1 per processor) and use Trace flag 1118 as a startup parameter (this alone gave me 10% in a recent performance lab). 
• Set autogrowth for the BizTalk databases

3.  Tune BizTalk Host Settings.  Here are the main ones I look for:

• CLR worker threads – to increase the number of processing threads
• For latency reduce the MaxReceive interval.  Be careful with this one – make sure that the polling interval is not set to a value lower than the execution time for any of your BTS_Deque stored procedures (there is one of these SPROCS per BizTalk host).  If this happens then BizTalk will overwhelm SQL by creating more connections in order to poll in time.
• Adjust the BizTalk config file max connection property if you are using HTTP

4.  Invest in a monitoring solution and continuously configure this as you learn about the system.

Q: Besides tuning hardware and deploying new infrastructure, there are ways that developers can directly impact (positively or negatively) the performance of their application through design choices they make.  What are some considerations that developers  should take into account with regards to performance when designing and building their applications?

A: The most important thing I think that developers can do is help drive a performance culture on their project.  In my opinion performance should be viewed as an essential quality attribute for any BizTalk project throughout its lifecycle. It is much more than a two week lab which is performed at the end of a project, or even worse not at all.   I think it is important to point out that many BizTalk applications are mission critical and downtime cannot be tolerated; in many cases, downtime of the solution can affect the liquidity of the company.  In my experience a proper performance engineering approach is not taken on many BizTalk projects.  Everyone is responsible for performance. Unfortunately, in many cases I have seen developers not realize until it is too late that the lack of consideration for performance from the beginning  resulted in decisions early in the lifecycle which really affected the potential performance of the system.  I would advise any developer who is considering using BizTalk to consider performance right from the beginning and to test continuously (not just two weeks before go-live).  If something that affects performance creeps into the build and is not detected for more than two weeks, it is likely that removing it will require a lot of engineering.  In many cases I see that developers do not want to invest in these assets due to the perceived “cost” of them.  I would use the word investment instead, for me investing in automated performance tests and metrics are assets that will save me a huge amount of time later on and help ensure the success of the project.

In terms of the application itself, ultimately bear in mind that everything that you add to an application will slow down the pure BizTalk engine.  Consider performance in everything that you do: e.g. within pipelines and Orchestrations minimize the use of XML document; instead use a streaming approach and XLANGMessage.  I would advise developers to invest in test assets which will enable them to continuously run performance tests and benchmarks .  I use BizUnit for my end to end functional testing.  I’ve found that using this in combination with the Visual Studio 2008 Load Test tools enables me to perform both automated functional testing and performance testing. Without a good baseline it is very difficult to determine whether a check-in has degraded or improved performance. BizTalk requires powerful hardware, therefore I would advise developers to invest in production quality machines at the beginning of a lifecycle. This will enable them to continuously run performance comparisons throughput the project.

The final thing that I’d like to see developers do is to train their operations/infrastructure team in BizTalk and how their application uses it.  Most infrastructure teams understand what SQL, Exchange and Active Directory does.  This helps them define their support processes and procedures.  In many cases infrastructure teams have no prior experience of BizTalk – so they treat and support it as a black box. By training them in the architecture of BizTalk Server, you will enable them to effectively tune and maintain the environment once it goes into production and also minimize the amount of support incidents which need to be escalated to the development team.  I know that this is something that you yourself have done Richard within your organization.

Q [stupid question]: Working at Microsoft, you have a level of credibility where folks believe the advice and information that you provide them.   This means that you could deliver them fake “facts” and they’d never know.  A few examples include:

• “Little known fact, if you add the registry key “Woodgate4” to HKLM\SOFTWARE\Microsoft\BizTalk Server\3.0,  Microsoft actually provides you three additional orchestration shapes.”
• “There are actually 13 editions of BizTalk Server 2009  including Home Premium and Web Edition.”

Ok, you get the point.  Give me a “fake fact” about technology that you could sell with a straight face.

A: If you hold down CTRL-SHIFT-P on startup, Windows Vista will load a custom version of Pac Man where your goal is to eat the evil Linux Ghosts.