Q: You recently authored a great book on BAM which went to a level of depth that no previous book had. Are there specific things you learned about BAM while writing the book? How would you suggest that BAM get greater mindshare among both BizTalk and .NET architects?
A: First, thanks and glad you found it to be useful. I must say, during the course of writing it, there were numerous concerns that all authors go through (is this too technical/not technical enough, are the examples practical, is it easy to follow, etc.). But because BAM has the limited adoption it does, there were additional concerns surrounding the diversity of real-world scenarios, better vs. best practices, and technical validation of some of the docs. As far as real-world experience goes, the shops that have implemented BAM have learned a lot in doing so, and are typically not that willing to share with the rest of the world their learning experiences. With technical validity, one of the more frustrating things in really diving into the nuances of subjects like the architecture of the WF interceptor, is that there is little if anything written about it (and therefore no points for reference). The documentation, well, one could say that it could be subject to reconsideration or so. I, regretfully, have a list of the various mistakes and issues I found in the MSDN docs for BAM. Perhaps it’s one of the major reasons BAM has the adoption it does. I think one of the other major issues is the tooling. There’s a utility (the TPE) for building what are, in essence, interceptor config files for BizTalk orchestrations. But if you want to build interceptor config files for WCF or WF, you have to manually code them. I think that’s a major oversight. It’s one of the things I’ve been working on in my spare time, and plan to post some free utilities and plug-ins to Visual Studio to a web site I’ve just set up, http://www.AllSystemsSO.com. I do thank, though, those who have written about BAM, like Jesus Rodriguez, for penning articles that kept BAM on the radar. Unfortunately there hasn’t been a single volume of information on BAM to date.
Specific things I learned about BAM – well, with .NET Reflector, I was able to pull apart the WF interceptor. If you take it apart, and examine how tracking profiles are implemented in the WF interceptor, how they map to BizTalk tracking points, and the common concepts (like persistence), it’s a really fascinating story. And if you read the book where I’m explaining it (chapter 9), you may be able to note a sense of wonder. It’s clear that the same team in CSD that wrote BizTalk wrote WF, and that many of the same constructs are shared between the two. But even more interesting are that certain irritations of the BizTalk developer, like the ability to manually set a persistence point and the pluggable persistence service, made their way into WF, but never back into BizTalk. It gave me pause, and made me think when Devs have asked of Microsoft “When will BizTalk’s orchestration engine support WF?” and Microsoft’s answer has been “It won’t, it will continue to use XLANGs,” perhaps a better question (and what they meant) was “when will BizTalk and XLANGs support all the additional features of WF.”
As for gaining greater mindshare, I wrote one of the chapters specifically along the lines of how BAM fits in your business. The goal of that chapter, and largely the book, was to defeat the notion that BAM is purely BizTalk-specific. It’s not. It’s connected systems-focused. It just so happens that it’s bundled with BizTalk. Yes, it’s been able to ride BizTalk’s coattails and, as such, be offered as free. But it’s a double-edged sword, as being packaged with BizTalk has really relegated BAM to BizTalk-only projects. I think if people had to pay for all the capabilities that BAM offers in a WCF and WF monitoring tool, it would clearly retail for $30k-50k.
If BAM is to gain greater mindshare, .NET and connected systems developers need to make it a tool of their arsenal, and not just BizTalk devs. VPs and Business Analysts need to realize that BAM isn’t a technology, but a practice. Architects need to have an end-to-end process management strategy in mind, including BI, BAM, Reporting, StreamInsight, and other Performance Monitoring tools.
RFID is a great vehicle for BAM to gain greater mindshare, but I think with Microsoft StreamInsight (because it’s being built by the SQL team), you’re going to see the unification of Business Activity Monitoring and Business Event Monitoring under the same umbrella. Personally, I’d really like to see the ESB Portal and Microsoft Services Engine all rolled up into a BAM suite of technologies alongside StreamInsight and RFID, and then segmented off of BizTalk (maybe that’s where “Dublin” is going?). I’d also like to see a Microsoft Monitoring Framework across apps and server products, but well, I don’t know how likely that is to happen. You have Tracing, Debugging, the Enterprise Framework, Systems Center, Logging Servers, Event Viewer, and PerfMon for systems. You have BAM, PerformancePoint, SSRS, Excel and Custom Apps for Enterprise (Business) Performance Monitoring. It’d be nice to see a common framework for KPIs, Measured Objectives, Activities, Events, Views, Portals, Dashboards, Scorecards, etc.
Q: You fill a role of lead/manager in your current job. Do you plan delivery of small BizTalk projects differently than large ones? That is, do you use different artifacts, methodology, team structure, etc. based on the scale of a BizTalk project? Is there a methodology that you have found very successful in delivering BizTalk projects on schedule?
A: Just to be clear, BizTalk isn’t the only technology I work on. I’ve actually been doing A LOT of SharePoint work in the last several years as it’s exploded, and a lot of other MS technologies, which was a big impetus for the “Integrating BAM with ____” section of the book.
So to that end, scaling the delivery of any project, regardless of technology, is key to its success. A lot of the variables of execution directly correlate to the variables of the engagement. At Avanade, we have a very mature methodology named Avanade Connected Methods (ACM) that provides a six-phase approach to all projects. The artifacts for each of those phases, and ultimately the deliverables, can then be scaled based upon timeline, resources, costs, and other constraints. It’s a really great model. As far as team structure, before any deal gets approved, it has to have an associated staffing model behind it that not only matches the skill sets of the individual tasks, but also of the project as a whole. There’s always that X factor, as well, of finding a team that not only normalizes rather quickly but also performs.
Is there a methodology that I’ve found to be successful in delivering projects on schedule? Yes. Effective communication. Set expectations up front, and keep managing them along the way. If you smell smoke, notify the client before there’s a fire. If you foresee obstacles in the path of your developers, knock them down and remove them so that they can walk your trail (e.g. no one likes paperwork, so do what you can to minimize administrivia so that developers have time to actually develop). If a task seems too big and too daunting, it usually is. Decomposition into smaller pieces and therefore smaller, more manageable deliverables is your friend – use it. No one wants to wait 9 months to get a system delivered. At a restaurant, if it took 4 hours to cook your meal, by the end, you would have lost your appetite. Keep the food coming out of the kitchen and the portions the right size, and you’ll keep your project sponsors hungry for the next course.
I think certain elements of these suggestions align with various industry-specific methodologies (Scrum focuses on regular, frequent communication; Agile focuses on less paperwork and more regular development time and interaction with the customer, etc.). But I don’t hold fast to any one of the industry methodologies. Each project is different.
Q: As a key contributor to the BizTalk R2 exam creation, you created questions used to gauge the knowledge of BizTalk developers. How do you craft questions (in both exams or job interviews) that test actual hands-on knowledge vs. book knowledge only?
A: I wholeheartedly believe that every Architect, whether BizTalk in focus or not, should participate in certification writing. Just one. There is such a great deal of work and focus on process as well as refinement. It pains me a great deal whenever I hear of cheating, or when I hear comments such as “certifications are useless.” As cliché as it may sound, a certification is just a destination. The real value is in the journey there. Some of the smartest, most talented people I’ve had the pleasure to work with don’t have a single certification. I’ve also met developers with several sets of letters after their names who eat the fish, but really haven’t taught themselves how to fish just yet.
That being said, to me, Microsoft, under the leadership of Gerry O’Brien, has taken the right steps by instituting the PRO-level exams for Microsoft technologies. Where Technology Specialist exams (MCTS) are more academic and conceptual in nature (“What is the method of this technology that solves the problem?”), the PRO-level exams are more applied and experiential in nature (“What is the best technology to solve the problem given that you know the strengths and limitations of each?”). Unfortunately, the BizTalk R2 exam was only a TS exam, and no PRO exam was ever green-lit.
As a result, the R2 exam ended up having somewhat of a mixture of both. The way the process works, a syllabus is created on various BizTalk subject areas, and a number of questions is allotted to each area. Certification writers then compose the questions based upon different aspects of the area.
When I write questions for an interview, I’m not so much interested in your experience (although that is important), but moreso your thought process in arriving to your answer. So you know what a Schema, a Map, a Pipeline, and an Orchestration do. You have memorized all of the functoids by group. You can list, in order, all of the WCF adapters and which bindings they support. That’s great and really admirable. But when was the last time your boss or your client asked you to do any of that? A real-world generic scenario is that you’ve got a large message coming into your Receive Port. BizTalk is running out of memory in processing it. What are some things you could do to remedy the situation? If you have done any schema work, you’d be able to tell me you could adjust the MaxOccurs attribute of the parent node. If you’ve done any pipeline work you’d be able to tell me that you’re able to de-batch messages in a pipeline as well into multiple single messages. If you’ve done Orchestrations, you know how a loop shape can iterate an XML document and publish the messages separately to the MessageBox and then subscribe using a different orchestration, or simply using a Call Shape to keep memory consumption low. If you’ve ever set up hosts, you know that the receive, processing, tracking, and sending of messages should be separate and distinct. Someone who does well in an interview with me demonstrates their strength by working through these different areas, explains that there are different things you could do, and therefore shows his or her strength and experience with the technology. I don’t think anyone can learn every aspect or feature of a product or technology any more. But with the right mindset, “problems” and “issues” just become small “challenges.”
Certification questions are a different breed, though. There are very strict rules as to how a question must be written:
- Does the item test how a task is being performed in the real-world scenario?
- Does the item contain text that is not necessary for a candidate to arrive at the correct answer?
- Are the correct answer choices 100% correct?
- Are the distracters 100% incorrect?
- Are the distracters non-fictitious, compelling, and possible to perform?
- Are the correct answers obvious to an unqualified candidate?
- Are the distracters obvious to an unqualified candidate?
- Does the code in the item stem and answer choices compile?
- Does the item map to the areas specified for testing?
- Does this item test what 80% of developers run into 80% of the time?
It’s really tough to write the questions, and honestly, you end up making little or nothing for all the time that goes in. No one is expected to score a perfect score, but again, the score is moreso a representation of how far into that journey you have traveled.
Q [stupid question]: It seems that the easiest way to goose blog traffic is to declare that something “is dead.” We’ve heard recently that “SOA is Dead”, “RSS is Dead”, “Michael Jackson is Dead”, “Relational Databases are Dead” and so on. What could you claim (and pretend to defend) is dead in order to get the technical community up in arms?
A: Wow, great question. The thing is, the inevitable answer deals in absolutes, and frankly, with technology, I find that absolutes are increasingly harder to nail down.
Perhaps the easiest claim to make, and one that may be supported by observations in the industry as of late, is that “Innovation on BizTalk is Dead.” We haven’t seen any new improvements really added to the core engine. Most of the development, from what I understand, is not done by the CSD team in Redmond. Most of the constituent elements and concepts have been decomposed into their own offerings within the .NET framework. BizTalk, in the context of “Dublin,” is being marketed as an “Integration Server” and touted only for its adapters. SharePoint development and the developer community has exploded where BizTalk development has contracted. And any new BizTalk product features are really “one-off” endeavors, like the ESB Toolkit or RFID mobile.
But like I said, I have a really hard time with that notion.
I’ve just finished performing some training (I’m an MCT) on SharePoint development and Commerce Server 2009. And while Commerce Server 2009 is still largely a COM/COM+ based product where .NET development then runs back through an Interop layer in order to support the legacy codebase, I gotta say, the fact that Commerce Server is being positioned with SharePoint is a really smart move. It’s something I’m seeing that’s breathing a lot of fresh air into Commerce Server adoptions because with shops that have a SharePoint Internet offering, and a need for eCommerce, the two marry quite nicely.
I think Innovation on BizTalk just needs some new life breathed into it. And I think there are a number of technologies on the horizon that offer that potential. Microsoft StreamInsight (“Project Orinoco”) has the potential to really take Activity and Event Monitoring to the next level by moving to ultra-low latency mode, and allowing for the inference of events. How cool would it be that you don’t have to create your BAM Activities, but instead, BAM infers the type of activity based upon correlating events: “It appears that someone has published a 50% off coupon code to your web site. Your profit margin in the BRE is set to a minimum of 30%. Based on this, I’m disabling the code.” The underpinnings to support this scenario are there with BAM, but it’s really up to the BAM Developer to identify the various exceptions that could potentially occur. CEP promotes the concept of inference of events.
The M modeling language for DSL, WCF and WF 4.0, Project Velocity, and a lot of other technologies could be either worked into BizTalk or bolted on. But then again, the cost of adding and/or re-writing with these technologies has to be weighed.
I’d like to see BAM in the Cloud, monitoring performance of business processes as it jumps outside the firewall, intra- and inter- data centers, and perhaps back in the firewall. Tell me who did what to my Activity or Event, where I’m losing money in interfacing inside my suppliers systems, who is eating up all my processing cycles in data centers, etc. I really look forward to the day when providing BAM metrics is standard to an SLA negotiation.
I’m optimistic that there are plenty of areas for further innovation on BizTalk and connected systems, so I’m not calling the Coroner just yet.
Thanks Jeff. If any readers have any “is dead” topics they wish to debate, feel free.
Ok, so now I have two distinct services which add orders for a particular type of event. Now, I want to expose a single external endpoint by which systems can place orders. I don’t want my service consumers to have to know my back end order processing system URLs, and would rather they have a single abstract endpoint which acts as a broker and routes messages around to their appropriate target. So, I created a new WCF Service web application. At this point, just for reference I have four projects in my solution. 
Alrighty then. First off, I removed the interface and service implementation files that automatically get added as part of this project type. We don’t need them. We are going to reference the existing service type (Routing Service) provided by WCF 4.0. Next, I went into the .svc file and changed the directive to point to the FULLY QUALIFIED path of the Routing Service. I didn’t capitalize those words in the last sentence just because I wanted to be annoying, but rather, because this is what threw me off when I first tried this back in December.
After the change is made, we can view the Configuration file associated with the WCF Test Client and see that our endpoint now refers to the Routing service. 
Coolio. Now, we can test. So I invoked the BuyTickets operation and first entered a “Sports” type ticket. 
Then, ALL I did was switch the EventType from “Sports” to “Concert” and the Routing service should now call the service which fronts the concert reservation service. 
There you have it. What’s nice here, is that if I added a new type of ticket to order, I could simply add a new back end service, update my Routing service filter table, and my service consumers don’t have to make a single change. Ah, the power of loose coupling. You all put up with these types of posts from me even though I almost never share my source code. Well, your patience has paid off. You can grab the full source of the project here. Knock yourselves out. 
Richard Seroter's Architecture Musings 