Richard Seroter's Architecture Musings

Author: Richard Seroter

  • Troubleshooting “Canceled Web Request”

    Recently, when calling web services from the BizTalk environment, we were seeing intermittent instances of the “WebException: The request was aborted: The request was canceled” error message in the application Event Log. This occurred mostly under heavy load situations, but could be duplicated even with fairly small load.

    If you search online for this exception, you’ll often see folks just say to “turn off keep-alives” which we all agreed was a cheap way to solve a issue while introducing performance problems. To dig further into why this connection to the WebLogic server was getting dropped, we actually began listening in on protocol communication using Wireshark. I started going bleary-eyed looking for ACK and FIN and everything else, so I went and applied .NET tracing to the BizTalk configuration file (btsntsvc.exe.config). The BizTalk documentation shows you how to set up System.Net logging in BizTalk (for more information on the settings, you can read this).

    My config is slightly different, but looks like this …

    <system.diagnostics>
    <sources>
      <source name="System.Net">
        <listeners>
          <add name="System.Net"/>
        </listeners>
      </source>
      <source name="System.Net.Sockets">
        <listeners>
          <add name="System.Net"/>
        </listeners>
      </source>
      <source name="System.Net.Cache">
        <listeners>
          <add name="System.Net"/>
        </listeners>
      </source>
    </sources>
    <switches>
      <add name="System.Net" value="Verbose" />
      <add name="System.Net.Sockets" value="Error" />
      <add name="System.Net.Cache"  value="Verbose" />
    </switches>
    <sharedListeners>
      <add name="System.Net"
           type="System.Diagnostics.TextWriterTraceListener"
           initializeData="c:\BizTalkTrace.log"   />
    </sharedListeners>
    <trace autoflush="true" />
  </system.diagnostics>

    What this yielded was a great log file containing a more readable format than reading straight network communication. Specifically, when the request was canceled message showed up in the Event Log, I jumped to the .NET trace log and found this message …
    A connection that was expected to be kept alive was closed by the server.

    So indeed, keep-alives were causing a problem. Much more useful than the message that pops up in the Event Log. When we did turn keep-alives off on the destination WebLogic server (just as a test), the problem went away. But, that couldn’t be our final solution. We finally discovered that the keep-alive timeouts were different on the .NET box (BizTalk) and the WebLogic server. WebLogic had a keep-alive of 30 seconds, while it appears that the .NET framework uses the same value for keep-alives as for service timeouts (e.g. 90 seconds). The Windows box was attempting to reuse a connection while the WebLogic box had disconnected it. So, we modified the WebLogic application by synchronizing the keep-alive timeout with the Windows/BizTalk box, and the problem went away complete.

    Now, we still get the responsible network connection pooling of keep-alives, while still solving the problem of canceled web requests.

    Technorati Tags:

  • Upcoming SOA and Business Process Conference

    Chris and Mike confirm the details of the upcoming SOA and Business Process conference in October 2007.

    I was fortunate enough to attend last year’s event, and would highly encourage folks to attend this year. It’s a great place to meet up in person with BizTalk community folks and receive a fair mix of high level strategy overview and deep technical content. It’s always fun to put a face with a particular BizTalk blogger. I usually find that my mental image of the person is woefully inaccurate. I have a picture of someone like Tomas Restrepo in my head, but watch, I’ll meet him in person and find out that he’s a 82 year old Korean woman with bionic legs.

    That said, I actually can’t attend this year, and am quite disappointed. My wife and I chose THAT week to be due with our first child. The nerve. Maybe I can convince Chris Romp to bring a cardboard cut-out of me so that I can be there in spirit.

    Technorati Tags:

  • BizTalk Sending Updated Version of Message to SOAP Recipients

    What happens to downstream SOAP recipients if the message sent from BizTalk is a different “version” than the original?

    Let’s assume I have an enterprise schema that represents my company’s employees (e.g. “Workforce”). BizTalk receives this object from SAP and fans it out to a variety of downstream systems (via SOAP). Because direct messaging with the SOAP adapter is occurring, a proxy class is needed to call the web service (vs. an orchestration). Because every subscriber service implements the same enterprise schema and WSDL, a single proxy class in BizTalk can be reused for each recipient.

    Using “wsdl.exe” I do code generation on the enterprise WSDL to build an interface object (using the wsdl /serverinterface command) that is implemented by the subscribing web service (thus ensuring that each subscriber respects the enterprise WSDL contract). I also use wsdl.exe to build the service proxy component that BizTalk requires to call the service. Finally, I use xsd.exe to build a separate class with the types represented in the enterprise schema.

    Now what if a department requests new fields to be added to this object? How will this affect each downstream subscriber? For significant changes (e.g. backwards breaking changes such as removal of required fields, changing node names), the namespace of the schema will be updated to reflect the change. This would force a rebuild and recompile of subscribers of this new message. For more minor changes, no namespace update will occur.

    We had a debate about how a .NET web service would handle the receipt of “unexpected” elements when the namespace has not been changed. That is, what if we just sent this new data to each subscriber without having them recompile their project with the latest auto-generated code (interface, proxy, types) from the enterprise schema/WSDL? Some folks thought that the .NET web service would reject the inbound message because it wouldn’t serialize the unknown types. I wasn’t 100% sure of that, so I ran a few tests.

    For this test, I added a new value to the auto-generated “types” class (which is used by the interface class and proxy class).

    I then build and GAC the updated proxy component. I also made sure that the subscriber web service still has the “old” auto-generated code.

    {New) Object used by BizTalk service proxy:

    {Old) Object used by subscriber web service:

    So, we’ve established that the web service knows nothing about “nickname”. If I add that field to my input document, and pass it in, route it to my subscriber port, what do you think happens? The first line of the web service writes a message to the event log, thus proving whether or not the service has successfully been called.

    I’ve turned on tcptrace so that I can ensure that “nickname” actually got transferred over the wire. Sure enough, the SOAP request contains my new field …

    Most importantly, an Event Log entry shows up, proving that my service was called with no problem.

    Interesting. So unexpected data elements are simply not serialized into the input object type, and no exception is thrown. I also tried using the “old” proxy class (e.g. without “nickname” in it) within BizTalk, and if schema validation is turned OFF, BizTalk also accepts the “extra” fields, but, since the “nickname” doesn’t exist in the proxy, does NOT send it over the wire, even though it was in the original XML message. Within the subscriber service, I could have serialized the object BACK into its XML format, and then applied an XML schema validation, and this would have raised a validation exception.

    Conclusion
    This is all good to know, but NOT a pattern or principle we will adopt. Instead, when changes are made to the enterprise schemas, we will create a new BizTalk map that “downgrades” the message in the subscriber send port. This way, we can gracefully update the subscribers at a later time, while still ensuring that they get the same message format today as they did yesterday. When the subscriber has recompiled their service with the latest auto-generated code, THEN we can remove the map from their send port and let them receive the newest elements.

    Technorati Tags: ,

  • [BizTalk] Interview Advice

    I recently completed another round of interviews for my company in the search for BizTalk consultants. Yet again, it was a fairly depressing experience. I offer a few humble tips to folks claiming to be BizTalk architects/developers.

    My first pet peeve is gigantic resumes. I know that headhunters often beef these things up, but if your resume has more pages than you have years of job experience, that’s a red flag. Brevity, people. If you’re still listing the top 14 accomplishments from your college internship in 1999, it’s time to reevaluate your resume building skills.

    In terms of the interview itself, I do NOT favor the types of questions that can be answered with one word or sentence (e.g. “tell me all the options when right-clicking a functoid” or “list me all the BizTalk adapters”). That doesn’t tell me anything about your skills. Tell me why you’d choose the HTTP adapter over the SOAP adapter. THAT’S a question.

    A few tips if you’re out there selling yourself as a BizTalk person …

    • Don’t list every possible subsystem/feature in BizTalk on your resume and claim experience. I simply don’t believe that EVERY person has used BAS. Come on. FYI, if you throw “Enterprise Single Sign On” on your resume, be ready for a question from me.
    • When you claim to be a BizTalk architect, and I ask you to explain the concept of “promoted values”, and you tell me that “I like to just promote any values in a schema that feel important”, the next sound you will hear is me stabbing myself in the neck.
    • When I see “experience building complicated orchestrations” on your resume, but my questions about “patterns” or “exception handling strategies” completely befuddle you, you’ve destroyed a piece of my soul.
    • When you tell me that you’ve participated in full lifecycle implementations (design –> test) on BizTalk projects, and I ask you what your next step is after requirements have been gathered by the business, and your answer is “start coding” … you’re not getting the job.
    • If you claim extensive experience with BizTalk application deployment, but you can’t tell me what artifacts may go into an MSI package, you’re not scoring any points.
    • While I appreciate honestly when I ask you for your BizTalk strengths and weaknesses, your chosen weakness should not be listed on your resume as an “expert with …”
    • If you tell me that you’ve spent significant time building BizTalk solutions that integrate with a database, and I ask how you poll for database records to kick off a process, the answer I’m looking for does not include “I build lots of custom data access classes.”
    • Finally, an interviewer can tell when you’re simply regurgitating an answer from a book or website. I want to hear answers in your own words. If you’ve stammered through the entire interview, but when I ask about the Business Rules Engine you provide an sweeping, poetic answer, I know you’re faking it.

    Sigh. I go into virtually every interview wanting to love the candidate, and roughly 75% of the time, I complete the interview sitting in a pool of my own tears. Any other tips you want to throw out there for BizTalk job candidates?

    Technorati Tags:

  • Automatically Generating Complex Types in BizTalk Schemas

    I’m currently teaching a “BizTalk Developer” class for my colleagues and as usual, I learned something I didn’t know before. Maybe it’s common knowledge, but it was new to me.

    Specifically, I never had figured out how to turn nodes in my schema into actual “complex types” to be used elsewhere. Let’s say I start with a “TrainingBooking” schema that validates messages from folks that are requesting training rooms for course delivery.

    We’ll also assume that I want to build a “room” object that other schemas may want to reuse. My “room” schema may look like this …

    Now I can import this “room” schema into my “booking” schema and reuse the defined types.

    However, the only “types” available from that schema when looking at the Data Structure Type in my “booking” schema is the root type. What if I want the “room details” node from my schema instead?

    What I just learned, was that if I go to my “room” schema, click my “room details” record node, and type in a value for the DataStructureType, then automagically, a “type” is created for me (see here for more details on creating global definitions).

    Sweet. Now if I go back to my “booking” schema, the new “RoomDetailsType” is accessible.

    I’ve had a lot of cases the past few months where I would have liked “real” complex types created, and now I know how. Learning is half the battle.

    Technorati Tags:

  • New “BizTalkHotRod” Newsletter Out

    Well, count me among the folks who thought that the BizTalk HotRod magazine would be a “one hit wonder.” I’m very happily proven wrong as there was a great new issue released this week. Some highlights include:

    • Deep look at various complicated BizTalk mapping scenarios and solutions
    • “Smack down” between a BizTalk and comparable WF/WCF solution with very thorough results
    • Look at BizTalk Server 2006 R2 EDI functionality
    • “Myth busting” about BizTalk distinguished fields
    • Throttling “how to”
    • Links to a comical amount of webcasts and resources

    Bravo to Todd and Sal (and all the contributors of this issue) for a very professional, must-read newsletter. Obviously a lot of work goes into these things, so make sure you go sign yourself up.

    Technorati Tags:

  • Choosing Between WF Rules and BizTalk Business Rules Engine

    If you’re still facing issues deciding which sort of workflow/rules technology from Microsoft to use (e.g. Windows Workflow vs. BizTalk), check out the latest well-written piece by Charles Young. He covers many of the specific differences to consider when deciding which Microsoft technology will work best for your given application.

    Technorati Tags: , ,

  • Preventing Stale Artifacts in BizTalk Application Packages

    A misunderstanding about how BizTalk builds MSI packages for applications caused me some recent heartache.

    Let’s say I have a BizTalk “application” with the following resources:

    As you can see, I’ve got a BizTalk assembly, a “standard” .NET assembly, binding file, text file, and virtual directory (for a web service). In my simple mind, I see the “Source Location” column, and assume that when I walk through the wizard to build an MSI package, BizTalk goes and grabs the file from that location and jams it into the final package.

    I was quite wrong. I learned this when I made updates to my web service and helper components, rebuilt my MSI package, deployed it, and learned to my horror that “old” components had been installed on the destination server. What gives?

    Whenever you add a “resource” to a BizTalk application (either by deploying your BizTalk library, or manually adding an artifact), it is added to a CAB file and stored in the database. In the BizTalkMgmtDb database there is an amazingly well-hidden table named adpl_sat which holds these uploaded resources.

    You can see in that picture that properties for each artifact are stored (e.g. GAC-ing components on install), and there’s also a column called “cabContent” which stores binary data. Updating artifacts on your file system does NOT mean they will get included in your MSI packages.

    So what’s a guy to do? Well, you may have noticed that for most resources, when you right-click and choose “Modify”, there is a button where you can “Refresh”. Now, instead of just refreshing the file from the spot designated in the “Source Location”, it asks you to browse to where the updated file resides. That seems a bit unnecessary, but whatever.

    So, that’s how you refresh MOST resources and make sure that your BizTalk application stays in sync with your developed artifacts. A big gotcha is, you CANNOT do this for virtual directories. I can’t seem to identify any way to refresh a modified web service short of removing the resource, and adding it back in. Big pain. Given how crucial web services are to many BizTalk applications, I would think that both adding them as resources (via UI vs. command line), and easily refreshing them, should be a priority for future releases.

    Technorati Tags:

  • Avoiding Service Timeouts In High Volume Orchestration Scenarios

    We were recently architecting a solution that involved BizTalk calling a synchronous web service from an orchestration in a high volume scenario. What happens if the web service takes a long time to complete? Do you run the risk of timeouts in orchestrations that hadn’t even had a chance to call the service yet?

    Let’s say you have an orchestration that calls a synchronous web service, like so …

    Assume that the downstream system (reached through the web service interface) cannot handle more than a few simultaneous connections. So, you can add the <add address = “*” maxconnection = “2” /> directive to your btsntsvc.exe.config file (actually, you should filter by IP address as to not affect the entire server).

    What happens if I have 20 simultaneous orchestrations? I’ve reduced the outbound SOAP threadpool to “2”, so do the orchestrations wait patiently, or fail if they don’t call the service in the allotted time? I started up 3 orchestrations, and called my service (which purposely “sleeps” for 60 seconds to simulate a LONG-running service). As you can see below, I have 3 running instances, but my destination server’s event log only shows the first 2 connections.

    The first two calls take 60 seconds, meaning the third message doesn’t call the service until 60 seconds have passed. You can see from my event log below that while the first 2 returned successfully, the third message/orchestration timed out. So, the “timeout” counter starts as soon as the send port is triggered, even if no threads are available.

    Now, what I found unexpected was the state of affairs after the timeouts. My next scenario involved dropping a larger batch size (13 messages) and predictably, I had 2 successes and 11 failures on the BizTalk server.

    HOWEVER, on my web server, the service actually got called 13 times! That is, the 11 messages that timed out (as far as BizTalk knows), actually went across the wire to the service. I added a unique key to each message just to be sure. It was interesting that after the BizTalk side timed out, all the queued up requests came over at once. So, if you have significant business logic in such a service, you’d want to make sure your orchestration had a compensating step. If you catch a timeout in the orchestration, there should be a compensating step to roll back any action that the service may have committed.

    So, how do you avoid this scenario? I tried a few things. First, I wondered if it was the orchestration itself starting the clock on the timeout when it detected a web port, so I removed the web port from the orchestration and used a “regular” port instead. No difference. It became crystal clear that the send port itself is starting the timeout clock, and even if no thread is available, the seconds are clicking by. I also considered using a singleton pattern to throttle the outbound calls, but didn’t love that idea.

    Finally, I came upon a solution that worked. If you turn on ordered delivery for the send port, then the send port isn’t called for a message until the previous one succeeds.

    This is one way to force throttling of the send port itself. To test this, I dropped 13 messages, and sure enough, the messages queued up in the send port, and no timeouts occurred.

    Even though the final orchestration didn’t get its service response back for nearly 13 minutes, it didn’t timeout.

    So, while not a fabulous solution, it IS a relatively clean way to make sure that timeouts don’t occur in high volume orchestration-to-service scenarios.

    Technorati Tags: ,

  • New BizTalk Whitepapers on SQL Adapter, Web Services

    A couple of useful BizTalk whitepapers released today. It’s good to see two once-dormant BizTalk blogs (first MartyWaz, now Doug from the CSD Customer Experience) wake up in the past couple weeks.

    The first paper, Best Practices for the SQL Adapter, covers how to properly write SQL receive location queries, deal with unsupported data types, handle annoying MSDTC errors, and resolve SQL Adapter Schema Wizard problems. It’s only 14 pages long, so you clearly have time to go read it right now. I’ll wait.

    Welcome back. The second paper, Sample Design Patterns for Consuming and Exposing Web Services from an Orchestration, explains in words and pictures how to best set up templates to consume and expose services from BizTalk orchestrations. It’s fairly short, but, should help people who are getting started with web services and orchestration.

    Technorati Tags: