So I spent a little time thinking about generic web service on ramps after reading Peter Kelcey’s post on the Microsoft ESB Guidance. The shockingly observant of you may recall that I wrote a post a few months back on how to build a BizTalk web service that accepted generic input as well.
I’m just not completely sold on the “one ramp to the bus” concept yet. Primarily because I worry about loosely typed service/object input. It’s nice that I can use the same snippet of code to publish a message to a service regardless of the message I’m using, but, ensuring that you’re passing the *right* data is arguably more important than the convenience of only having a few web services in the infrastructure. Now that we have such mature, powerful web service management offerings from companies like SOA Software, the physical number of services seems less of a concern.
All that said, I was having dinner with some techie buddies the other night and I raised this issue. We agreed that being able to validate your input PRIOR to sending the message to the generic service would be a way to mitigate this concern. So, I thought I’d try and build a simple “validation service” with BizTalk using orchestration and pipelines.
I started with a very basic couple of schemas. What I want to do is provide a simple service that take in any XML input, and returns back any validation errors.
Next comes the pipeline itself. In my Disassemble stage, I’m using the XML Disassembler with each available schema defined in the Document Schemas property. I set Validate Document to false here, and then used an XML Validator pipeline component later on to do the actual schema validation.
To start with, this is all I need to test. I wanted to test 4 different scenarios:
- Invalid data types (e.g. pass in string when int is expected)
- Missing nodes
- Adding additional records even though maxOccurs equals 1
- Passing in a message where no corresponding schema exists
I think that covers the basics of validation for now. So I built a pure messaging solution (FILE receive location, with FILE send port) and applied my custom receive pipeline to the inbound receive location. As expected, the result of each of the above scenarios was a suspended message. So my pipeline works. Now, I wanted to call this pipeline from an orchestration so that I would have a synchronous service that gracefully captured exceptions.
So, in my orchestration I referenced Microsoft.XLANGS.Pipelines.dll and Microsoft.BizTalk.Pipeline.dll. The process receives a document in (of type XmlDocument), and then within an atomic transaction (contained within a larger long-running transaction that catches exceptions), I called into my pipeline component. This is new functionality in BizTalk Server 2006. The code in my Expression Shape is:
RcvPipeOutMsgs = Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(typeof(Microsoft.Demo.Blog.ValidationSvc.Rcv_ValidateOnRampMsg), OnRampInput);
The RcvPipeOutMsgs is a variable of type Microsoft.XLANGS.Pipeline.ReceivePipelineOutputMessages, you see that I use the typeof my pipeline component, and the OnRampInput is the inbound message of type XmlDocument. The whole orchestration looks like this:
You can see that I catch any exceptions, set status variables, and then construct a “response” message to the service caller. I then walked through the Web Services Publishing Wizard (after changing my operation name from Operation_1 to PostDocToValidate) and built a web service that takes in any XML, and returns a common “response” message.
I then built a simple WinForm to call the service, passing in a variety of inputs. If I pass in a valid message, I get a “success” message. If I pass in a message with invalid content (scenarios 1-3 above), I get an error:
Looks good. HOWEVER, if I pass in a garbage message (that is, no schema exists), I get this:
I’m calling the exact same pipeline that I did in my “pure messaging” solution which DID raise an error for garbage messages, but when calling this pipeline from an orchestration, the pipeline isn’t raising an exception. I have tried about 37 different combinations of pipeline components, property settings, message types (tried switching from XML input to string) and I can’t see to get an error to occur.
So, there you have a 3/4 validation service. However, without properly handling bad messages, it’s not entirely useful. Thoughts? I really wanted to use the standard BizTalk components to validate (e.g. out of the box pipeline components), instead of using any sort of cool code-based solution. Anything I might be missing here?
Richard Seroter's Architecture Musings 
Why did you elect to not validate against a schema type on your XML Disassembler pipeline component? Would that throw an error?
Also, what happens if you try to utilize this pipeline in a messaging-only setup using the pipeline on your receive port? Does it error then?
During one of the pipeline variations I used just the XML Disassembler, and got the same non-error.
If I do messaging only, I DO get the exception raised. So, I know that the pipeline can detect the absense of a schema, but for some reason, when called from an orchestration, this is supressed.
That’s what I figured it would do. Are there options to invoking a pipeline from orchestration (I haven’t tried and my VM is broken right now) that are false by default but when BizTalk invokes them from messaging they’re enabled? Something like raising an exception when the schema validation component fails?
Would it be possible to use a custom validation pipeline component that *would* complain? I’m just throwing out ideas for how to accomplish this, though you did say you were trying this without code. =)
Richard,
Why do you feel it is important that the “messaging system” know anything about the message? It is commonly accepted that you should transform from native to common as soon as possible, and from common to native as late as possible, so that the message spends most of it’s time in the common format. Is it not a valid argument that only the producer and consumers of the data really care about the specific type? Moreover, isn’t it a valid argument that until the message is consumed, it doesn’t have any context, and therefore could potentially be treated an infinite number of ways; the consumer, or the producer actually apply the context, it is then that the data takes on meaning, for that context. Messaging systems add their own metadata (generally) to messages, so they can perform their required functions, but why would they care about the message data itself. I offer that strong typing should occur at the service boundaries.
BTW. Good to see that you’ve continued your committment to the community, even after moving on. I wish you well in your endeavors.
Hey Curt, I was waiting to hear from you guys.
I find it important that my service has an actual, meaningful contract. If I look at the basic tenets of exposing a service, I need to define a contract that explains the details of the service. Otherwise there are no clear instructions as to how to use the service, what encoding or security restrictions exist (or what security holes have been opened), possible responses, etc. Agreed that the bus itself shouldn’t care too much about the types it processes, but I want to prevent garbage or invalid messages from hitting the bus in the first place. Hence, a contract. I wouldn’t write a component that took in a “string” consisting of an XML document and was called “DoSomething(string xml)” because it’s confusing and lazy. I personally like my service signatures to be complete and clear.
Again, back to the ESB, sure, once a message is in the bus it gets fanned out all over, converting formats and types along the way. But I’m just not in love with having my ENTRY into the bus being a “loosey goosey” (Anti-Pattern: http://msdn2.microsoft.com/en-us/library/ms954638.aspx) service contract. I think there are certain cases where this pattern is ok and isn’t too objectionable, but I wouldn’t want to sacrifice straightforward architecture for developer/administrative simplicity.
Hi,
I found your post very interesting and I was wondering if there is any chance you can pass along some code…
That would be more than appreciated.
Regards,
D
Hey Dan,
Anything in particular you’re looking for that I didn’t include in the post itself?
Well what I am looking for is indeed some sort of validation service but that return the entire validation errors, does not stop on the first error… Please, can you point me maybe into some resources on the web? Thanks,Dan
You could consider using the BRE to validate a message for business conditions, and build up a list of exceptions encountered.