Interview Series: Four Questions With … Ewan Fairweather

Q: Whether someone has just purchased BizTalk Server, or is migrating an existing environment, appropriate solution sizing is critical.   What are the key questions you ask customers (or consider yourself) when determining how best to right size a BizTalk farm (OS, hardware, database, etc)?

A: I’ll start with the numbers that I use and then go through the thought process I use to scope the size of environment I need when I run performance labs.  The number of messages that you can process on a BizTalk Group depends on a lot of factors (machine characteristics, adapters, etc.).  Therefore the best way to size a BizTalk solution remains to do a POC. However if this is not possible, here are the numbers that I can and do use to make sure I am in the right ballpark.  These numbers are derived from our internal test results. The BizTalk machines were Dual Proc Dual Core, 4 GB memory and SQL was Dual Proc Quad Core with 8 GB of memory:

Single Server Messaging Performance

1. A simple small messaging  scenario can achieve 715 messages a second (WSHTTP WCF One Way Messaging Scenario using 2KB messages and passthru pipelines)
2. Optimizing the transport takes it to ~850 messages a second. Utilizing NetTCP provided us with an approx  ~18% gain.

The scale factor that I use is approximately 1.5 per BizTalk Server. So in this scenario going to 2 servers I would expect ~1000-1200 messages per second. At a certain point adding additional BizTalk Servers is going to cause the MessageBox SQL Server to become a bottleneck. To alleviate this Multiple MessageBox’s can then be added. In practice going past 4 or 5 MessageBox’s the returns begin to diminish.

Stage of the Project

The first thing I need to determine is the stage of the project.  If I am coming in at the beginning of the project, I will always seek to understand the customers business problem first, as it is only once I know this that I can ascertain their requirements. Once I understand a customer’s requirements the first question, I ask myself is “is BizTalk the right solution for this customers problem”.  BizTalk is a fantastic product but is not a universal panacea for all problems.  I strongly believe that positioning the right technology to solve the problem a customer has is my number one job, even if this means not using BizTalk.  A common question I am hearing (along with every other BizTalk person) is “when should I use BizTalk over Dublin and vice versa”.  Now answering that would be a full article in itself so rather than attempt it here I will refer to Charles Young’s very good article on the topic here.  I also use the numbers mentioned above to determine whether there system requirements are realistic.

Assuming that BizTalk is the right solution and will fulfill their requirements I then need to understand the key characteristics of the system, specifically I’m interested in the following:

1.  Message flow through the system. 

Now I know that there are often many tens of message flows through a system. For the majority of systems I’ve worked,  a much smaller subset of these tends to account for a large proportion of the load and hence have the most impact of perceived performance of the system.  I’ve found that identifying these and focusing on them is key.  For example, I recently worked with an customer in Europe to test their BizTalk system which was handling the back-end processing for their new Internet Bank.  In this case, over 80% of all requests were for the “Summary of Accounts view” (providing a consolidated view of all bank accounts).  In this scenario users are unlikely to wait a long time when they first log into the online bank therefore optimizing this and any directly message flows should be the key priority.    Once I’ve identified these flows, I’m primarily interested in the features of BizTalk that are being used.  Is messaging used heavily? Orchestration? Business Rules Engine? Is tracking required, either out of the box or BAM? I’m also trying to understand what external system are involved and how many calls are made from Orchestration.  Each of these features has some performance “cost” associated with it, my job as a Ranger is to work with the customer to ensure they get the functionality that they need at the minimum possible cost in terms of performance . In most cases I will put together a Visio diagram of the main message flows and clarify that these are correct with all the Developers of the system.

2.  Size/Format/Distribution of the messages

Message size and format is very important regarding BizTalk performance.  Processing binary files is expensive in terms of performance because often the message body cannot be compressed in the same way that XML can.   It is important to determine how big the messages will be and their distribution.  This is particularly important for large batches of messages that in many systems occur once a week.  I’ve seen many customers present me with extremely complex spreadsheets illustrating each and every single message type that will be processed in the system.  I think it is important to abstract to the appropriate level of detail otherwise dealing with and reproducing this in a lab becomes an impossible task. I typically ask customers to put together a table as per below with the constraint that it must be simple enough to fit on a single PowerPoint slide.

Response Type	Size	% of Traffic
Small	8413 bytes	20%
Medium	16998 bytes	60%
Large	52128 bytes	20%

3.  Production Traffic Can This be Replayed

Ideally in any performance testing or scoping scenario, I want to be able to replay actual data that will be processed in production.  Especially when the Rules Engine is used this is important to validate correct functionality.  Testing with a small subset of test data is better than nothing, but it is my experience that using production data will identify issues in system design before they get to production which is good for everybody.

4.  Performance Goals

Clear quantifiable goals are a must for anyone serious about BizTalk! Without these there is no quantifiable way to  judge the effectiveness of the system.  In short, performance goals are essential part of project success criteria. Good goals should state any constraints and should cover throughput and latency and any other relevant factors as well as how you will measure them. I’ve included an example below:

     Orchestration scenario

         Throughput :250,000 calls within 8 hours

                            ~9 messages/sec sustainable

          Latency:    < 3 seconds for  99% of all response messages

5.  Type of Environment

How many applications are present on the system – is this BizTalk group going to be processing a single application or is it going to be a centralized environment?

Q: You’ve done a great job capturing BizTalk performance tuning metrics and providing benchmarks for the results.   The sheer volume of options can seem intimidating, so give me only four modifications you would want all BizTalk owners to make in their environment.

A: I decided to cheat a bit on this answer and break my four modifications into areas: hardware, sql config, BizTalk config and monitoring.

1.  Invest in the right hardware – gigabit networking, fast storage (SAN or 15K local SQL disks), modern fast processing machines.

2.  Optimize SQL Server configuration for BizTalk. Including:

• Data and log file separation
• Create separate Temp DB data files (1 per processor) and use Trace flag 1118 as a startup parameter (this alone gave me 10% in a recent performance lab). 
• Set autogrowth for the BizTalk databases

3.  Tune BizTalk Host Settings.  Here are the main ones I look for:

• CLR worker threads – to increase the number of processing threads
• For latency reduce the MaxReceive interval.  Be careful with this one – make sure that the polling interval is not set to a value lower than the execution time for any of your BTS_Deque stored procedures (there is one of these SPROCS per BizTalk host).  If this happens then BizTalk will overwhelm SQL by creating more connections in order to poll in time.
• Adjust the BizTalk config file max connection property if you are using HTTP

4.  Invest in a monitoring solution and continuously configure this as you learn about the system.

Q: Besides tuning hardware and deploying new infrastructure, there are ways that developers can directly impact (positively or negatively) the performance of their application through design choices they make.  What are some considerations that developers  should take into account with regards to performance when designing and building their applications?

A: The most important thing I think that developers can do is help drive a performance culture on their project.  In my opinion performance should be viewed as an essential quality attribute for any BizTalk project throughout its lifecycle. It is much more than a two week lab which is performed at the end of a project, or even worse not at all.   I think it is important to point out that many BizTalk applications are mission critical and downtime cannot be tolerated; in many cases, downtime of the solution can affect the liquidity of the company.  In my experience a proper performance engineering approach is not taken on many BizTalk projects.  Everyone is responsible for performance. Unfortunately, in many cases I have seen developers not realize until it is too late that the lack of consideration for performance from the beginning  resulted in decisions early in the lifecycle which really affected the potential performance of the system.  I would advise any developer who is considering using BizTalk to consider performance right from the beginning and to test continuously (not just two weeks before go-live).  If something that affects performance creeps into the build and is not detected for more than two weeks, it is likely that removing it will require a lot of engineering.  In many cases I see that developers do not want to invest in these assets due to the perceived “cost” of them.  I would use the word investment instead, for me investing in automated performance tests and metrics are assets that will save me a huge amount of time later on and help ensure the success of the project.

In terms of the application itself, ultimately bear in mind that everything that you add to an application will slow down the pure BizTalk engine.  Consider performance in everything that you do: e.g. within pipelines and Orchestrations minimize the use of XML document; instead use a streaming approach and XLANGMessage.  I would advise developers to invest in test assets which will enable them to continuously run performance tests and benchmarks .  I use BizUnit for my end to end functional testing.  I’ve found that using this in combination with the Visual Studio 2008 Load Test tools enables me to perform both automated functional testing and performance testing. Without a good baseline it is very difficult to determine whether a check-in has degraded or improved performance. BizTalk requires powerful hardware, therefore I would advise developers to invest in production quality machines at the beginning of a lifecycle. This will enable them to continuously run performance comparisons throughput the project.

The final thing that I’d like to see developers do is to train their operations/infrastructure team in BizTalk and how their application uses it.  Most infrastructure teams understand what SQL, Exchange and Active Directory does.  This helps them define their support processes and procedures.  In many cases infrastructure teams have no prior experience of BizTalk – so they treat and support it as a black box. By training them in the architecture of BizTalk Server, you will enable them to effectively tune and maintain the environment once it goes into production and also minimize the amount of support incidents which need to be escalated to the development team.  I know that this is something that you yourself have done Richard within your organization.

Q [stupid question]: Working at Microsoft, you have a level of credibility where folks believe the advice and information that you provide them.   This means that you could deliver them fake “facts” and they’d never know.  A few examples include:

• “Little known fact, if you add the registry key “Woodgate4” to HKLM\SOFTWARE\Microsoft\BizTalk Server\3.0,  Microsoft actually provides you three additional orchestration shapes.”
• “There are actually 13 editions of BizTalk Server 2009  including Home Premium and Web Edition.”

Ok, you get the point.  Give me a “fake fact” about technology that you could sell with a straight face.

A: If you hold down CTRL-SHIFT-P on startup, Windows Vista will load a custom version of Pac Man where your goal is to eat the evil Linux Ghosts.