Richard Seroter's Architecture Musings

Author: Richard Seroter

  • Building Enterprise Mashups using RSSBus: Part III

    In the first two posts of this series, I looked at how to aggregate data from multiple sources and mash them up into a single data entity that could be consumed by an RSS client.  In this post, I will show off the new SOAP Connector from RSSBus.

    Earlier in this series, I talked about mashing up data residing in databases, Excel workbooks and existing web services.  In my scenario, I have an existing web service that returns a set of master data about our customers.  This includes contact details and which sales reps are assigned to this customer.  Let’s see how I can go about calling this service through RSSBus.

    Building an SOAP Feed

    First of all, let’s take a look at what parameters are available to the new RSSBus SOAP Connector.  The default properties that this connector needs are: URL, web method, and method URI (SOAP action).  However, there are a generous amount of additional, optional parameters such as parameter declaration, specific XML nodes to return, SOAP header, credentials, proxy server details and more.

    At the beginning of my RSSBus *.rsb file (which generates the XML feed), I specify the name of the feed, and, call out a specific input parameter (“customerid”) that the feed will require.

    Next, I set the properties that I want to pass to the connector.  Specifically, I identify the web service URL, method name, SOAP action, input parameter, and the place where I want to log all inbound requests and outbound responses.

    Now I can get a bit fancy.  I can pull out a particular node of the response, and work just with that.  Here, I dig into the XML service response and indicate that I only want the “Customer” node to be included in the result stream.  After that, I call out a series of XPath statements that point to the individual target nodes within that “Customer” node.  So, the final result stream will only contain these target nodes.

    Here I call the web method, and take the response values and put them into a new “cms” (“Customer Master System”) namespace with friendlier node names.  Note that the values returned by the SOAP connector are named after the XPath used to locate them.  For example, an XPath of “/Name/OrganizationName/FullName” would result in a SOAP Connector response element named “Name_OrganizationName_FullName.”  As you can imagine, the names for deep XPath statements could get quite unwieldy.

    If I make a request to this feed, I get back a nice clean result set.

    Now, I have one additional web service that I need to call from RSSBus.  If you recall from part I of this series, I need a way to know WHICH systems a particular contact resides in.  My company has a “object registry service” that stores contacts along with a pointer to the systems (and keys) that know something about that contact.  So, I can use this service to identify which feeds I need to call in order to get a complete picture of my contact.

    This RSSBus script takes in the name of the contact to find, calls my ORS (“object registry service”) service, and returns the systems that this person resides in.  In the resulting feed below, you can see that the ORS service found three records for this contact, and provides the name of the system (and primary key identifier) for each one.

    What’s Next?

    We’re close now.  I have feeds for a database, Excel workbook, Google web query, and two SOAP queries.  All that remains is to create the single feed that mashes up these system feeds and returns the single data result.

    Stay tuned for the exciting conclusion.

    Technorati Tags: ,

  • Building Enterprise Mashups using RSSBus: Part II

    In the previous post, I laid out a data visibility problem and proposed using RSSBus to build an enterprise mashup that inflates a single data entity whose attributes reside in multiple disparate systems.

    Before a mashup can be built, we actually need the source data in a ready-to-mashup format.   For the mashup I am building, my “contact” data resides in 3 different repositories:

    • Database containing the interactions we’ve had with a particular contact
    • Web service talking to a CRM system which holds core contact information
    • Excel spreadsheet containing external reference data such as the contact’s public web page and blog

    On top of this, my mashup will also return a Google result set based on the contact’s first and last name.  If available, I also want to retrieve the latest information from the contact’s personal blog (if they have one).

    In this post, I will show how to create the feeds for the database, Excel spreadsheet, and Google query.

    Building an Excel Feed

    My first data source is an Excel spreadsheet.  The Excel Connector provided by RSSBus has a variety of operations that let you add rows to sheets, list worksheets in a workbook, create new workbooks, and get data out of an existing spreadsheet.

    For our case, we used the excelGet operation that accepts the file path of the workbook, and which sheet to pull data from.  A simple test can be executed right from the connector page.

    The result of this query (formatted in HTML) looks like this:

    Notice how the Excel data comes back using an “excel” namespace prefix.

    In my case, I don’t want to return the contents of the entire workbook, but rather, only the record for an individual contact.  So, from this Connector page, I can choose to create a feed out of my sample query, and then I can modify the RSBScript to filter my results, and, to put my result set into a different namespace than “excel.”

    At the top of my new feed, I outline the title of the feed, and, create a new required input parameter named “contactid.”  Input parameters are passed to the feed via the querystring.

    Next, I need to set the parameters needed by the Excel Connector.  You may recall that we set the parameters when we tested the Connector earlier.

    Now comes the meat of the feed.  Here I “call” the Excel operation, do an “equals” check to see if the row in the spreadsheet is for the contact with the designated contact ID.  If I find such a row, then I create a new “myitem” entity and populate this hash with the values returned by the Connector and sitting in an “excel” namespace.  Finally, I “push” this item to the response stream.

    So what does this completed feed look like?  Below you can see our feed item containing the nodes in a “reference” namespace.  I now have a valid RSS feed that monitors an Excel workbook.  Hooray for me!

    Building a Database Feed

    Now, let’s take an existing SQL Server database and return some results as RSS.  In my scenario, I have a table for all our contacts, and another table with all the interactions we’ve had with that customer (e.g. lunch, office visit, speaker invitation).  The RSSBus SQL Server Connector has a wide range of operations available which perform database inserts, updates, deletes, as well as stored procedure calls, and schema queries for tables, views and more.

    This feed starts much the same as the last one with a title and description.  I’ve also added a required input parameter for the contact ID stored in the database.  Next I have to set the parameters (connection and query) needed by the sqlQuery operation.

    Note that most connector operations have a wide range of optional parameters.  For the sqlQuery operation, these optional parameters include things like “maxrows” and “timeout.”

    Now I need to call the operation.  Like the feed above, this feed takes things that come back in the “sql” namespace and put it into an “interactions” namespace.  Be aware that the “push” statement pushes EACH returned row as a separate feed item.

    What does my response look like?  The image below shows how I returned three items in this feed; each represents a different interaction with this contact.

    Now I have two feeds based on existing data repositories, and didn’t have to make a single change to those applications to support their RSS-ification.

    Building a Google Feed

    The final feed we’ll look at here is public internet search for our selected contact.  The Google search results should come back in an RSS format that can be consumed by my mashup feed.

    My feed takes in two required parameters: “firstname” and “lastname.”   Next, I need the two critical parameters for the Google gSearchWeb operation.  I first must pass in a valid Google API token (you’ll need to acquire one), and, the search string.

    Now I call the operation, and push each result out.

    Neato.  I can call Google on the fly and make custom queries based on my contact.

    What’s Next?

    Here we saw how easy it is to build the RSSBus script necessary to expose RSS feeds from systems that don’t actually speak RSS.

    Next, we’ll see how to work with new RSSBus SOAP connector to query our CRM system, AND, to query my “object registry service” which returns all the system primary keys related to my contact.  After that, we’ll see how to mash all these feeds up, and return a single “contact” entity to the RSS client.

    Technorati Tags: ,

  • Building Enterprise Mashups using RSSBus: Part I

    I’ve been a fan of RSSBus from /n software for some time. A few weeks ago our Executive Director / Chief Architect / Technology Overlord recently asked me to build a real, live enterprise mashup application to demonstrate for our IT leadership group. Our goal was to show that RSSBus could be used to quickly and efficiently aggregate data in compelling new ways.   In the next few posts, I’m going to walk through our use case, and how I built a solution to solve this.

    Before getting into the “what” and “how”, I want to first say that the “how I built a solution” line above isn’t entirely true.  The folks at RSSBus actually built most of the solution for me, and all I did was tweak and customize it a bit.  If you ever get a chance to work with rock stars Ralph James, Amit Sharma or Tom Hearn, jump at it.  I’ll also note that I am working with a pre-release version 2.0 of RSSBus that has new features such as a SOAP connector.

    The Problem

    As with most large organizations, we have a multiple systems and repositories that contain different aspects of the same data entity.  A “contact” at my company has some attributes stored in our ERP system, some in custom built systems, and some more in COTS applications.  Maintaining a true system of record is difficult in our environment and trying to keep enterprise data repositories in sync is no small task.

    Some questions I want answers to …

    • How does a sales rep heading out on a physician visit get a full 360 degree view of the customer that shows EVERYTHING we know about this person?
    • How do I allow people to be proactively notified of any changes to a given sales region or key contact?
    • How do I easily accommodate new data sources (e.g. COTS application, public internet blog or website)?
    • Where can I apply the appropriate security measures to make sure that sensitive contact details are only available for those allowed to see them?  For instance, sales folks may not be allowed to see what clinical trials a physician is participating in, while scientists are not allowed to see marketing activities that a physician has been involved with.

    This category of problem can also extend to “complex event processing” scenarios where you want to be able to perform intelligent surveillance on related events that span systems.   However, the problem I’m addressing for us at this moment has to do with data aggregation, not event aggregation.

    A Solution

    One valid solution to this problem is to use XML feeds (RSS / Atom) from source systems and aggregate them to return a single, complete view of the target data entity.

    Why XML feeds instead of a new database repository?  Our reasons include:

    • Subscription model using XML feeds provides users with a great ability to discover, organize and monitor data that sits in multiple places
    • Consumption experience NOT dictated by data generator as end users can choose to eat this data in their feed reader (e.g. Outlook 2007), Excel, SharePoint or custom application
    • Provides a virtual, on-demand data aggregation with minimal changes needed in source systems
    • The pub/sub nature of XML feeds gives users more “sensors” into key data elements in an organization and allows them to receive and act on data in a more timely fashion
    • XML feeds allow an alternate level of service where a query does not have to return real time data or in an immediate fashion

    Once I have a feed for each system that stores “contact” details, I want to mash them all up and return a single XML feed that shows an entity whose data comes from all types of data stores.

    Now, how do I know which systems this “contact” exists in? At my company, we have a “object registry service” that our applications use to both publish and query enterprise objects.  For instance, we have CRM applications which send “insert” and “update” commands to this service when contact data has changed.  This service is responsible for performing data cleansing on inbound data, and matching inbound objects to existing objects.  The “Richard Seroter” contact inserted by System A should be matched to the “Richard L Seroter” that was already inserted by System B.  What this service stores is only enough information to perform this matching, and, the originating system and primary key.  So, the point is, I can query this service for “Richard Seroter”, and get back all records matching this query, AND, which system (and ID) stores information about this handsome character.

    One other wrinkle.  Clearly, many (most?) COTS and custom applications do NOT offer RSS feeds for their underlying data.   So how do I go down this route of XML feeds with systems that don’t natively “talk” RSS? This is where RSSBus comes in.

    What is RSSBus?

    RSSBus is a lightweight, completely web-based platform for building and publishing XML feeds out of a wide variety of source systems.  It’s a service bus of sorts that takes advantage of the loose contract of RSS to expose and aggregate feeds into interesting business services.

    RSSBus uses an impressive array of “connectors” to sources such as Amazon Web Services, PayPal, Microsoft CRM, MySQL databases, Federal Express, Twitter, FTP, LDAP, BizTalk Server and much more.  This means that you can create a feed out of a file directory, or use XML feeds to create new LDAP accounts.   Endless possibilities.

    The RSSBus Administration Console can be used to browse connectors, and then use a visual wizard to prototype and generate XML feeds.  You also have full access to the robust RSBScript language which is actually used to generate the raw feeds and optionally, the RSBTemplates which present feed data in an HTML format.  It’s almost a misnomer to call the product RSSBus since data can be emitted not only as RSS, but also ATOM, XLS, CSV, JSON, HTML and SOAP.

    Why is RSSBus a good solution for my problem?  Some reasons include:

    • Minimal programming needed to connect to a wide range of mixed platform technologies
    • Strong ability to combine feeds into a single information source
    • Deep scripting and function library for working with and formatting feed data
    • Nice support for feed caching and feed security
    • Can separate data (XML) from presentation logic

    RSSBus is NOT a complete XML feed management solution by itself.  That is, RSSBus doesn’t have the full feed discovery and management that an Enterprise Server (NGES) from Newsgator is.  However, note that NGES now plays with RSSBus so that the powerful feeds generated by RSSBus can be managed by NGES.

    What’s Next?

    In the next set of posts, I’ll look at how I exposed individual RSS feeds from a mix of data sources including Microsoft Excel spreadsheets, databases, web services, and Google queries.  After that, I’ll show you how to mash up the individual feeds into a single entity.  Then, I MAY demonstrate how to apply security and caching aspects to the mashup.

    Technorati Tags: ,

  • Goodbye BizTalk 2006 R3, Hello BizTalk 2009

    Good communication today about not only the name change, but more importantly, the updated roadmap for BizTalk Server (read Q & A here, see roadmap here, and see BizTalk home page here, see Steve Martin’s announcement here).

    For me, the most important things communicated were:

    • greater clarification on what the Oslo release means to BizTalk Server
    • specific features in BizTalk Server 2009
    • a commitment to a continued 2+ year release rhythm of BizTalk releases
    • recognition of the types of new features we’d like to see added to BizTalk (low latency support, developer enhancements, more platform integration)

    This is a great lead in to the upcoming PDC, and a smart move to reassure BizTalk customers who may have been a bit wary of what was coming down the pipe.

    Technorati Tags: ,

  • Interview Series: Four Questions With … Alan Smith

    Last month I started a new blog series where I interview a different “connected systems” thought leader each month and get some insight into their experiences and thinking in our space.  Our first victim/subject was Tomas Restrepo. We continue the series by chatting with everyone’s favorite Swedish BizTalker, Alan Smith.

    Q: You spend a healthy amount of time facilitating the MSDN BizTalk R2 forums. What are some common assumptions or misunderstandings about BizTalk Server that you frequently see questions about?

    A: There are quite a few people asking about using BizTalk for heavy database integration, taking flat files, inserting the data in databases and processing it. SQL Server Integration Services (SSIS) does a much better job than BizTalk at this, and is worth looking at in those scenarios. BizTalk still has its uses for that type of work, but is limited be performance. The worst case I saw was a client who had a daily batch that took 36 hours to process using BizTalk, and about 15 minutes using SSIS. On another project I worked on they had used BizTalk for all the message based integration, and SSIS for the data batching, and it worked really well.

    Another common one is with mapping, there’s a lot of scenarios where replacing the map with custom XSLT is the easiest solution to solve the problem. People can spend days torturing themselves with functoids without getting the problem solved. A lot of questions appear on the forums where custom XSLT is the best solution. This picture says it all. With some scenarios the mapper is great, but with others custom XSLT can be easier to write and debug, and results in much more efficient code. The XSLT IntelliSense and debugging features in Visual Studio 2005 make it very easy to code and test. I really should get a webcast on custom XSLT published, it’s been on my to-do list for a while.

    Lastly, there seem to be a lot of people who are using BizTalk Server 2006 R2, and use the SOAP adapter for consuming and publishing web services. If you can, try using the WCFBasicHttpAdpater for this, as it has so much more options for configuration, and it seems to work much better.

    Q: You’re probably most famous for producing the invaluable “Bloggers Guide to BizTalk” that many of us used in those early BizTalk 2004 days as we harvested knowledge from everywhere possible. What’s the status of this resource?

    A: Back in the day, when BizTalk 2004 was in beta, the blogs and the forums were where you leant about how stuff worked. The early BizTalk documentation was not that great, there were no books available, and I had several people tell me they used the guide more than the documentation. Now things have changed and we have pretty good documentation, good courses, and some great books, so the guide is not as relevant as it once was. I have just released a Version 2.0 of the guide, and will make updates if people are downloading it a lot.

    I’ve already started work on “The Bloggers Guide to Oslo”, there’s not that much content yet, as anyone who knows anything is sworn to secrecy though an NDA, but after PDC in November this should change. I think after the details of Oslo have been announced publicly at PDC it will be similar to the early days of BizTalk 2004. There should be some great contributions from the development community and this will probably be the best source of information for those learning the technology. I’m looking forward to it already…

    I’ll be hosting both of these guides, at http://bloggersguides.net/. I recently launched this site to provide a permanent location for the guides along with webcasts and samples related to those technologies.

    Q: As you’ve grown in your understanding of connected systems, are there common patterns that you frequently reuse, and conversely, patterns (or anti-patterns) that you purposely avoid?

    A: Most of the patterns and practices I use tend to be related to the development process. Automated deployment, automated testing, and a good project structure and naming conventions should be used in every project. As for messaging patterns, the “Enterprise Integration Patterns” book is the best resource, BizTalk Server provides quite a few of those patterns out of the box, and makes it very easy to implement other patterns.

    One thing I would avoid would be passing every message through an orchestration. Some projects I have seen use this as a pattern, and it adds complexity to the solution, and an overhead in processing. Another one to avoid is publishing services using BizTalk that will be consumed by user interfaces. In some scenarios it can work, but you have to be aware of the latency issues and how to tune BizTalk for low latency.

    Q: Nearly everyone has that silly or embarrassing song that they sing along with in their car with the windows rolled up. I will freely admit that when Bon Jovi’s “Living on a Prayer” comes on, I get stuck on stupid. Expose your dark secrets by telling us which song or musician inexplicably gets your juices flowing.

    A: Right now it’s probably the James Zabiela Essential Mix from Glastonbury. In my misspent youth I used to attend the Glastonbury festival, can’t remember too much except it was muddy and there were flashing lights, but this mix helps to bring back the vibe.

    Technorati Tags: ,

  • What Does an Architect Do?

    Great pointer from Mike Walker yesterday to an IASA blog post highlighting their Architect Taxonomy and what it means to be an architect.  Reading things like this are always a reminder to me that I’m not remotely great at my job yet.

    The author of the IASA post, Paul Preiss, first links to the list of “Certified Architect” requirements from Microsoft which focus on five broad competency areas:

    • Leadership.  Are you providing thought leadership, mentoring and able to build consensus for important ideas and standards?
    • Communication.  Can you effectively share your ideas both orally and in the written word, and do so for a variety of audiences? 
    • Organizational dynamics.  Do you have a feel for your company’s key decision makers and can you work through these organizational structures to get things done?
    • Strategy.  Can your knowledge of technology help your organizational position itself favorably for the future while also applying the frameworks and principles to make you successful today?
    • Process and Tactics.  Are you able to navigate the project lifecycle and efficiently work through system requirements, design, prototyping, documentation and deployment?
    • Technology Breadth.   Do you have a grasp on a wide range of technologies and concepts that may comprise a complete organizational solution? 
    • Technology Depth.  Are you a thought leader within your organization on specific topics?

    The IASA taxonomy is even more in depth than what Microsoft provided.   They bunch up their expected skill set into five buckets:

    • IT Environment
    • Business-Technology Strategy
    • Design Skills
    • Quality Attributes
    • Human Dynamics

    I really liked their breakdown of each category and the specifics listed under each.  After calling out the core skill buckets, they go into the expectations for a number of different flavors of architect: software architect, infrastructure architect, and business architect.

    Very useful to read, if nothing else, than to help plan a roadmap for things to strengthen in the upcoming years.  Also, this is a great cheat-sheet for coming up with questions during an architecture interview!

    Technorati Tags: Architecture

  • Odds and Ends

    A couple things to share on this glorious Wednesday. 

    First, one of our BizTalk developers came across an odd scenario, and I just learned that everyone’s favorite Dahan (Yossi Dahan) wrote about it a couple years ago.  Our issue was that an orchestration needed to consume a service with a multi-part input, but the complex type parameter wouldn’t show up as message for mapping.  After a quick poke around, I see that Yossi explained how multi-part messages that have a mix of complex parts and simple type parts cannot be used in a map.  I tried it out, and sure enough, if you create a multi-part orchestration message that does not have all parts set as schema types, then that message isn’t available as a map input or output.  Odd and unappreciated.

    Secondly, my company is looking for a simple AS2 solution for integration with a business partner, and was evaluating some available choices.  If (and when) we formally upgrade to BizTalk 2006 R2, we’ll obviously get that capability for free.  In the meantime, we’re also considering either the full AS2 adapter from our buddies at /n software, or the free single-partner AS2 solution also offered by /n software.  I only mention all this because I got a healthy chuckle out of the unconventional FAQ page on the http://www.freeas2.com site.  I’m easily amused, so this was right up my alley.

    That’s all.  Carry on.

    Technorati Tags: BizTalk

  • Impact of Database Availability on BizTalk Web Services

    My buddy Victor asked me the other day about the relationship between IIS and the BizTalk databases.  That is, if we restart the SQL Server service or server, what happens to messages that are still submitted to the BizTalk web services on an active IIS server?

    So, I put together a really quick application where I tested four scenarios: downstream host unavailable, IIS unavailable, receive location offline, and SQL Server unavailable.

    Also, to legitimately gauge service behavior, I exposed both classic ASMX services and WCF services for my BizTalk application.  Both services were built as one-way HTTP services hosted in IIS.  The published data is then routed to a single FILE send port via message-type routing.

    Scenario: Processing Host is Unavailable

    For this scenario, I simply disabled the in-process host that runs the send port subscribing to messages published by the services.

    Result: Messages are published with no problem, and everything is queued up until the in-process host comes online.  No message loss and no errors to the service callers.

    Scenario: IIS is Unavailable

    Here I turned off the IIS website hosting the services.

    Result: As expected, both the ASMX and WCF services returned errors to the client application.  The ASMX service returned an error saying:

    error: System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:80

    The WCF service returned the following error:

    error: System.ServiceModel.EndpointNotFoundException: Could not connect to http://myserver/Blog.Biztalk.AvailabilityTestWCF/ContractService.svc. TCP error code 10061: No connection could be made because the target machine actively refused it 192.168.131.65:80.  —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 192.168.131.65:80

    So the client gets an error, no message is submitted to BizTalk by either service, and the client will be expected to try again later.

    Scenario: Receive Location is Offline

    Now I’ve turned off the actual receive locations.  The website is up and running, but the BizTalk receive locations aren’t listening for the inbound messages.

    Result:  The ASMX service returns a success message (HTTP 202), even though no message is published to BizTalk.  There is an error in the System Event log stating:

    The Messaging Engine could not find the receive location for URI:”/Blog.Biztalk.AvailabilityTest/ContractService.asmx”.\
    Please verify the receive location exists and is enabled.

    However, the client does NOT get an error even though no message was published (or suspended) by BizTalk.

    The WCF service returns an HTTP error and the following message to the client:

    error: System.ServiceModel.ServiceActivationException: The requested service, ‘http://myserver/Blog.Biztalk.AvailabilityTestWCF/ContractService.svc’ could not be activated. See the server’s diagnostic trace logs for more information.

    In this case again, no message is published, BUT, at least the client knows that a problem occurred.  Much better than the ASMX behavior.

    Scenario: SQL Server is Offline

    In this case, I’ve shut down the SQL Server service and the in-process hosts that are running.

    Result: The ASMX service continued to return HTTP success messages, even though it could not publish to the MessageBox.  The IsolatedHost (which runs the Message Agent) can’t connect, but the client isn’t told this.

    The WCF service, however, returns the same error it did on the previous scenario.  So it did not publish the message either, but, again, it returned a proper exception to the client.

    Looking at the IIS logs, I wanted to confirm the service response.  For the ASMX service call when the database was offline, I can see the following entry:

    POST /Blog.Biztalk.AvailabilityTest/ContractService.asmx – 80 – 127.0.0.1 202 0 0

    Notice the HTTP 202 returned to the client.  The next entry in the log file represents my call to the WCF service while the database was still down:

    POST /Blog.Biztalk.AvailabilityTestWCF/ContractService.svc – 80 – 192.168.131.65 – 500 0 0

    Notice the HTTP 500 error which represents an internal server error returned to the caller.

    Summary

    So, we can conclude that ASMX services do a lousy job of reporting what actually happens after it tries to publish messages to the BizTalk bus.  Unless the IIS server is explicitly taken down during database server maintenance or restarts, you run the real risk of losing messages without the client being aware of it.  For WCF services, we see much better handling of message publishing problems.   This is probably due to the fact that the BizTalk WCF service host relies heavily on the BizTalk configuration database and receive location availability to complete its operations.  While it still can’t save the inbound requests, it at least tells the caller that something went wrong.

    Anyone else have different experiences than the ones I demonstrated above?

    Technorati Tags: , BizTalk

  • Trying *Real* Contract First Development With BizTalk Server

    A while back on my old MSDN blog, I demonstrated the concept of “contract first” development in BizTalk through the publishing of schema-only web services using the Web Services Publishing Wizard.  However, Paul Petrov rightly pointed out later that my summary didn’t truly reflect a contract-first development style.

    Recently my manager had asked me about contract-first development in WCF, and casually asked if we had ever identified that pattern for BizTalk-based development.  So, I thought I’d revisit this topic, but start with the WSDL this time.  I’m in the UK this week on business, so what better use of my depressing awake-way-to-early mornings than writing BizTalk posts?

    So like Paul had mentioned in his post, a true service contract isn’t just the schema, but contains all sort of characteristics that may often be found in a WSDL file.  In many cases (including my company), a service is designed first using tools that generate WSDLs and XSDs. Then, those artifacts are shared with service developers who build services that either conform to that service (if exposing an endpoint) or consume it from other applications.

    I’ll start with a simple WSDL file that contains a schema definition and a request/response operation called HelloWorld.  The schema contains a few constraints such as maxOccurs and minOccurs, and a length restriction on one of the fields.

    What I’d like to do is have BizTalk consume the WSDL, and then generate a service that respects that WSDL.  How does BizTalk eat a WSDL?  Through the use of the dark and mysterious BPEL Import BizTalk project type.

    After choosing this project type, a wizard pops up and asks you for the artifacts that make up the service.  In my case, I just pointed it to the WSDL which had an embedded schema definition.

    After an “import succeeded” message, I’m left with a new project and three files which represent my schema (and base types), and an orchestration that includes the auto-generated port types and schemas.

    For simplicity’s sake, I’ll just build out the provided orchestration, with the goal of exposing it as a web service.  First, I add a new configured port to the orchestration design surface, careful to choose the generated port type provided to me.

    I’m not sure why, but my generated multi-part message isn’t configured right, and I had to manually choose the correct schema type for the message part.

    Next, I built two messages (request and response) which used the generated multi-part message types.

    Finally, I added send/receive shapes (and a construct) in order to complete the simple orchestration.

    I’ll show what happens when using the ASMX Web Publishing Wizard, but first, let’s be forward thinking and use the WCF Service Publishing Wizard.  I chose a WCF-BasicHTTP endpoint with metadata so that I can inspect the WSDL generated by my service and compare it against the original.  You’ll notice that the “service name” of the service is a combination of the orchestration type name and namespace, and, the “service port” is the name of the orchestration port.  Feel free to change those.

    I then had to change the target namespace value to reflect the target namespace I used in the original WSDL file.

    After completing the wizard (and build a receive location so that my service could be hosted and activated), I compared the WSDL generated by BizTalk with my original one.  While all of the schema attributes were successfully preserved (including original restrictions), the rest of the base WSDL attributes did not transfer.  Specifically, things like the SOAP action and service name were different, not to mention all the other attribute names.

    I went back and repeated this process with the ASMX Web Publishing Wizard, and there were differences.  First, the wizard actually kept my original target namespace (probably by reading the Module XML Target Namespace property of the generated orchestration) and also allowed manual choice of “bare” or “wrapped” services.  The actual generated WSDL wasn’t much better than the WCF wizard (SOAPAction still not right), and worse, the schema definition was stripped of important restriction characteristics.  This is a known problem, but, annoying nonetheless.

    At this point, you can argue that this is a moo point since I can take advantage of the ExternalMetadataLocation property on the Metadata Behavior in my generated configuration file.  What this does is allow me to point to any WSDL and use IT as the external facing contract definition.  This doesn’t change my service implementation, but, would allow me to use the original WSDL file.  If I set that configuration attribute, then browsing my BizTalk-generated service’s metadata returns the base WSDL.

    One of the key things to remember here is that the SOAPAction value you use is the value set in the BizTalk “MethodName” context attribute.  This value is used to match inbound messages to their orchestration subscription (when bound to a SOAP port).  If these don’t line up, you get “subscription not found” errors when you call this service.  So, if I generate my WCF contract using the original WSDL, and submit that message to my WCF endpoint, the message context looks like this:

    And remember that my orchestration port’s generated “operation” property was “HelloWorld” and thus my MessageBox subscription is:

    So, if you plan on using an external WSDL, make sure you line these values up.  Also note that the orchestration port’s “operation” property doesn’t accept an “http://xxx” style value, so pick something else 😉

    I plan on including this topic in the upcoming book, but didn’t feel like squirreling it away until then.   It’s an interesting topic, and I’d be curious as to other’s attempts and getting BizTalk to conform to existing service contracts.

    Technorati Tags: , BizTalk

  • Interview Series: Four Questions With … Tomas Restrepo

    There are a plethora of great technologists in the “connected systems” space, and I thought it would be fun to interview a different one each month.  These are short, four question interviews where I ask about experiences with technology.  The last question will always be a fairly stupid, silly question that might only amuse me.  So be it.  I used to do these sorts of interviews when I wrote newsletters for Avanade and Microsoft, so if I happen to reuse a previously asked stupid question, it’s because I liked it, and assume that most of my current readers never saw those old newsletters.  I’m a cheater like that.

    To start things off, let’s have a chat with Tomas Restrepo.  Blogger extraordinaire , Microsoft MVP, and all around good guy.

    Q: Tomas, you’ve consistently been out in front of many Connected Systems technologies such as BizTalk Server and WCF.  What Microsoft technologies are on your “to do” list, why, and how do you plan to learn them?

    A:  That’s really a tough question to answer. There’s just so much stuff coming out of Redmond these days, and, to be honest, it’s still to early to tell yet how much of it is going to “stick” and what might be abandoned down the road in favor of something else.

    Sometimes learning a new technology in depth can be quite time consuming, so you want to be careful when choosing what to invest your time in. What I’m currently trying to do is follow a few rules:

    • Try to be aware of “what’s out there” and at least know what it does and what it is good for.
    • Figure out which things are interesting enough (or show enough potential) to dig into a bit deeper. Not enough to master them, but enough to know the big concepts behind them and how to apply them.
      These are stuff you play with a little bit, and would consider good enough to start a POC with them if the need arises and then dig into them big time when you start a project with them.
    • Stuff that’s really important that you want to really spend a lot of time tinkering with them and mastering them.

    I think there are some interesting things out there worth keeping an eye on. For example, I don’t do much web development these days, but if I had to, I’d immediate dig deeper into the ASP.NET MVC framework. I’m already familiar with Castle’s monorail and somewhat with rails and other similar technologies, so it should be easier to get started.

    I’m also definitely looking forward to some of the stuff in Oslo. Obviously the core framework and WCF stuff is going to be pretty interesting there. I’ve been keeping an eye on the cloud services (BizTalk Services) stuff as well, but I’m really waiting there for a project idea that really demands those capabilities before spending more time with them.

    Certainly there’s a lot of things that will be coming out in the next one-two years such as the updates to the big products (SQL, Visual Studio and so on), and those will get their fair share of time when the time comes.

    Q: In your experience, what is your criteria for deciding between either using (a) a broker such as BizTalk Server between systems or (b) directly consuming interfaces/services between systems?

    A:  I think this is one case where there are both technical and non-technical reasons for making this decision.

    On the technical side I very much try to start questioning whether any kind of mediation is required/desired and whether BizTalk is the right kind of tool for that job. In particularly, I’d look into the latency and performance requirements, the protocols being used for the services and the amount of data that needs to be transferred between systems.

    Part of this is looking to see if, for example, the project is in a low-latency scenario or perhaps if it’s really a set of bulk data processes more suitable to something like SSIS.

    Another thing to look for is whether you need the kind of capabilities that BizTalk offers. For example, would the interface be better served with Pub/Sub support? Would the Pub/Sub support in BizTalk be enough, or does it require heavier duty pub/sub with thousands of subscribers and possibly transient (non-persistent) subscriptions?

    BizTalk has some great support for some kind of messaging scenarios, but it also has limitations that can constrain your solution heavily. Sometimes you can clobber your project needs into BizTalk by extending the product in different ways (thank goodness for its extensibility!), but it’s not always the best option available.

    On the non-technical side, a few aspects that matter are: Does the client already own a BizTalk license they can use? If not, can the project/client budget take assume that cost? Sometimes it can be negotiated, but other times it’s just not an option. Besides the raw cost of licensing, there are of course knowledge aspects, like, does the company have people already familiar with the technology?

    In other words, I’ve found that the non-technical aspects of the use/don’t use BizTalk aren’t too different from the kind of aspects you’d consider for acquisition of any new technology. That said, BizTalk does pose it’s own challenges on an organization because of it’s complexity.

    That said, I do try to be very careful to avoid looking at the world with technology-tainted glasses. It’s important to approach a new project with an open mind and figure out what the best technology to solve the client needs are, instead of starting with a given technology (BizTalk, in this case) and try to cram the project requirements into it whatever the cost. Sometimes the non-technical aspects of the project might suggest/impose a technology decision on you, but even in that case it’s important to take a step back, breath deeply and make sure it’s the best option available to you.

    Q: You’ve been working with a variety of non-Microsoft technologies lately.  What are some of the interoperability considerations you’ve come across recently?  Share any “gotchas” you’ve encountered while getting different platforms to play nicely together.

    A:  No matter how you look at it, interoperability isn’t easy, and you can’t take it for granted. It’s something you need to keep very much in check every step of the way and verify it time and time again.

    Certainly Web Services (of both the SOAP and REST varieties) have helped here somewhat, but not all interoperability issues come from the lower-level transport protocols; sometimes the application / service interface design can have a bit impact on interoperability.

    One rule I try to follow is to design for interoperability. For example, if I’m designing a new service interface, I want to know who my clients are going to be; what technology they are going to be using and what constrains they might have.

    Sometimes, the best option you can take is to stick to the basics: simple works. That’s actually one of the beauties of REST architectures. As long as you’ve got an XML parser and an HTTP client, you’re in business, and HTTP is known well enough (and has such a good tooling around it for development and diagnosis) that it really helps a lot.

    Basic SOAP is also pretty good nowadays, if used correctly. The WS-* specs, like WS-Security and friends are pretty important in some scenarios. They are published standards, yes, but getting interoperability isn’t as easy as with plain SOAP and rest, because they are very complex specifications.

    For example, if you’re using message-level encryption, and you run into trouble, then raw protocol level interception won’t help you at all to diagnose the issue; you really need tooling support on your SOAP stack for this (WCF’s is pretty good).

    Once you get into using X.509 certificates for encryption/signing or even just for raw authentication, things can get hairy pretty quickly. Mostly this is because a lot of people don’t quite understand how X.509 certificate validation works, and common problems arise from invalid certificates, certificates installed to the wrong store, or just because someone forgot to deploy the entire certificate trust chain.

    By themselves, they are not though problems to solve, but diagnosing them can be very challenging at times because the tooling isn’t always very good at reporting the right reasons for failure. Anyone who has been stuck with a “Error validating server identity” kind of error can attest to that 🙂

    WS-Security specs also have the pose another challenge, and it’s that there are multiple versions of those specs out there, and sometimes you find yourself using one version with your partner using another. You have to be very careful in specifying and validating the right protocol version.

    Q [stupid question]:  Everyone has that one secret pet peeve that makes them crazy.  I’ll admit that mine is “mysterious stickiness.”  I shudder at the thought of touching a surface and coming away with a unwanted adhesive.  Ugh.  Tell us, what is something that really drives you nuts?

    A: Cockroaches. I hate cockroaches. They give me the creeps.

    Seriously speaking, though, I think that my main problem is that I can be very impatient about the little things. Stuff like getting short delays from things can drive me crazy (a stuck keyboard or mouse can really go out of this world).

    Hope you all find these interviews a bit interesting or at least mildly amusing.

    Technorati Tags: ,