Author: Richard Seroter

More MVP Summit Thoughts

Yesterday, me and the other BizTalk (and Connected Systems) MVPs began participating in a software design review of Oslo. Not much we can talk about, but needless to say, the Microsoft team’s been busy. Interesting stuff so far.

On top of the great group of folks I had met up with the day before, it was more of the same yesterday. Got a chance to chat with Jesus Rodriguez, Jon Flanders, Winson Woo, Yossi Dahan, Saravana Kumar, Kent Weare, Matt Meleski and Eric Stott. It’s quite beneficial to get the insights and perspectives of the folks who use the software vs. just those who build it.

Technorati Tags: BizTalk

April 16, 2008
MVP Summit Thoughts

Here at the Microsoft MVP Summit in Seattle and it’s been fun bumping into old friends and putting names with faces. Had a chance to hang out with Scott Colestock, Charles Young, Brian Loesgen, Stephen Thomas, Tim Rayburn, Alan Smith, Paul Somers, Tomas Restrepo and Bill Chesnut. It’s pretty wild to be around some of the smartest BizTalk folks on the planet.

Today’s MVP Summit sessions were intentionally unfocused and meant to be conversations among folks with similar technology interests. I spent my first session in a WCF roundtable moderated by Michele Leroux Bustamante where I walked away with lots of ideas and items to research (WCF null channels, security token services, etc).

I next sat in on a “patterns and practices” discussion, and quickly realized how fortunate BizTalk developers are to “get” many concepts that other developers may struggle with. I asked the group how they handled asynchronous design patterns, and that spawned a 30 minute discussion on the challenges of asynchronous designs (specifically on the UI) and how many developers are leery to embrace it. There was further discussion the challenges around exception management, logging and transactions. Charles Young and I were chatting and agreed that many of these areas which plague “regular” developers are things that a BizTalk developer has encountered and conquered years ago. For instance, any reasonable BizTalk developer understands the value of asynchronous messaging, and is adept and dealing with callbacks and correlation. For exception handling, the BizTalk engine itself handles exceptions fairly well by suspending instances, without the developer explicitly telling it to please not discard the instance or messages.

If I take a step back and ignore any features I WISH BizTalk had, I still have to admit that the evolution from BizTalk 2002 to 2004 is one of the greatest software engineering feats that Redmond has produced. Think about all the well constructed architectural changes ( messaging engine, adapters, .NET assemblies) and new features (SSO, BRE, BAM, VS.NET integration) that were included in a single release cycle, and still hold up today. Impressive, well thought out stuff. Not that everything’s perfect, or close to it, but BizTalk developers are exposed to a lot of broad architectural principles that benefit them even in non-BizTalk solution scenarios. And yes, the Kool-Aid is flowing in Seattle this week.

Technorati Tags: BizTalk, WCF

April 14, 2008
Using the BizTalk Adapter Pack to Load Oracle Data into InfoPath 2007 Forms

One of the glaring omissions from InfoPath 2007 is integration with non SQL Server data repositories. For retrieving reference data (query data), your only choice is SQL Server. That’s lousy. For submitting data, your only choice is SQL Server or Microsoft Access, and, you can only submit to a database when using the thick client version of InfoPath. That is, you can’t do a direct database submission on the InfoPath Forms Services version of a form. This also sucks. SharePoint isn’t much better, as the SharePoint Designer integration with ODBC data sources (such as Oracle) is clumsy at best, and in our experience, not really functional.

But, complaining gets me nowhere. So, how can I cleanly interface between InfoPath and Oracle, which is the overwhelming database preference in my current environment? What I’ll show here is how to use the BizTalk Adapter Pack to generate the service stubs I need to query and publish to Oracle databases. I’ll be publishing another article on TopXML.com that digs deeply into the BizTalk Adapter Pack, so I won’t go into great depth here on what it is.   But the short description is: the BizTalk Adapter Pack contains Line of Business (LOB) adapters that use a common WCF infrastructure which can be consumed by WCF clients, ADO.NET clients (for ETL scenarios), BizTalk clients (for robust workflow, EAI scenarios), or basic SOAP clients (for single operation scenarios). I’m building an example for the last option.

First off, I need data in my Oracle database tables. If you are a regular Visual Studio.NET user, you know that you can access ODBC data sources (such as Oracle) using the built-in Server Explorer. This works fine for data entry, but if you want true control over the database, you need more. One really nice option (from Oracle itself) is the Oracle Developer Tools for Visual Studio.   This is a nice plug in for VS.NET that gives you most of what you need for Oracle design tasks.

Lots of great stuff in there. For my purposes, I needed a new table to start with.

Then I had to fill in some initial data.

Now we can actually build the service which interacts with this table. First, I created a new “WCF Service Library” project in Visual Studio.NET. From there, once the BizTalk Adapter Pack is installed, I can select Add Adapter Service Reference from the “Add” menu.

This opens a window where I can select my connection criteria, browse objects in my repository, and select the operations I want. The first thing to do is configure the URI. After choosing which binding to use (in my case, the OracleDbBinding), I have to set the DataSourceName value. This is the identifier in my Oracle tnsnames.ora file. We no longer use System DSN entries to connect to Oracle and instead put database details in the tnsnames.ora file. After setting the data source, I have to put the username and password for my Oracle database.

Once all my connection details are set, I can connect to my database. Now, I can search (*new feature alert*) and browse the database tables, views, functions, packages and procedures.

After choosing my database table operation (“Select”) and closing out the window, I get an auto-generated application configuration file, and, a class containing all the interfaces, contracts and proxy code needed to consume this database service. Now, as stated earlier, I want to take this WCF service, and expose it out for basic SOAP clients (i.e. InfoPath/SharePoint). Before building the web service proxy around this WCF client, I want to add a brand new service interface. I don’t want to directly expose the Adapter Pack interface, but rather, put a simpler interface out to my SOAP clients.

So, in this WCF Service Library project, I added a new class containing a WCF ServiceContract and OperationContract, as well as the service which implements those contracts and calls my auto-generated Adapter Pack proxy.

So I have a friendly service interface which abstracts some of the BizTalk Adapter Pack details, but calls the auto-generated Oracle Adapter bits when it’s operation is invoked.

Now, I need to put this service into a web-ready container. All I have now is the library project. So, I created a new “Web Site” project of type “WCF Service.” I don’t need much from this service, so the underlying “cs” file can be deleted. Then, after adding a reference to my WCF Service Library project, I have to modify the Service.svc file so that it points at my implementation class (“Seroter.Demo.BizTalkAdapterPack.CustomerSvcLibrary.CustomerQuery”) for it’s “Service” attribute.    After this, all that remains is to set up the web.config file to both see my Oracle Adapter endpoint (as a service) and expose new endpoints for my SOAP clients. After copying the information from the auto-generated app.config file (that should still be in the WCF Service Library project) into my web.config, I can launch the Service Configuration Editor to make the remaining web.config changes. Specifically, I need to add a new service (and endpoint), and provide a metadata behavior so that clients can query the service metadata.

So now I have a client endpoint (which uses the BizTalk Adapter Pack Oracle binding), and a new service with an HTTP endpoint that basic SOAP clients can use. After creating a new virtual directory in IIS 6.0, and pointing to the WCF Service project, I now have a ready-to-go database access service.

From InfoPath 2007, I can now call and consume this service to pull reference data from Oracle. Specifically, I created a new data connection, pointed to the IIS service, and chose the operation I needed.

Now I can drag the repeating results node to the InfoPath form and create a table. If I preview the form, the service gets called, and the results are displayed.

That was easy! Took about a dozen lines of code and a configuration file setup.   In a later post, I’ll show how to use this service to populate data forms in SharePoint as well.

Technorati Tags: WCF, InfoPath

April 4, 2008

Article Series on BizTalk and WCF: Part VI, Publishing Advanced Services Patterns

UPDATE: I have since moved these articles to my own blog and they can be found here.

I just posted another article for TopXML.com as part of my series on BizTalk + WCF. This post digs into exposing WCF services out of BizTalk that utilize security features, MTOM attachments, and transactions.

Topics include: message-based security, transport security, custom role based authorization, MTOM attachments, and how to send messages to BizTalk as part of a client transaction. I observed some odd behavior with regards to MTOM where the “request” on the request/response port correctly attached the binary content as a MIME part, but the “response” message kept the binary string inside the XML payload itself.

Series Summary
BizTalk and WCF: Part I, Operation Patterns	Get the source code!
BizTalk and WCF: Part II, Security Patterns
BizTalk and WCF: Part III, Transaction Patterns
BizTalk and WCF: Part IV, Attachment Patterns
BizTalk and WCF: Part V, Publishing Operations Patterns	Get the source code!
BizTalk and WCF: Part VI, Publishing Advanced Service Patterns
BizTalk and WCF: Part VII, About the BizTalk Adapter Pack	Get the source code!
BizTalk and WCF: Part VIII, BizTalk Adapter Pack Service Model Patterns
BizTalk and WCF: Part IX, BizTalk Adapter Pack BizTalk Patterns

Technorati Tags: BizTalk, WCF

April 1, 2008

New Collection of WCF Security Task Guidance

If you’re looking to dig a bit deeper into WCF security, and specifically how to perform particular tasks, go check out J.D. Meier’s blog for a link list of new “how tos” and videos recently posted to CodePlex.

Specifically, you’ll find some useful demonstrations for creating temporary certificates for message-based security, some interesting looks at impersonation, using SQL Role Providers for authentication, and more.

Technorati Tags: WCF

March 27, 2008

BizTalk Environment Migration Checklist

My company’s “standard operating procedure” for BizTalk Server doesn’t call out the specific requirements to deploy among environments (development to test, test to production, etc), so I’m trying to help the team get those articulated. Here’s my first stab at a checklist that should be followed for BizTalk application migration between environments. I don’t want the list to be abusive, but want it to be as comprehensible as possible. Any feedback is appreciated.

Migrate from Local Development to Shared Development
	Task	Comments
1.	Code review executed by BizTalk team	Refer to BizTalk Code Review guidelines
2.	Consistent system artifact naming in place	System DSNs, application configuration settings
3.	Server access request approved	BizTalk Administrator rights ok
4.	User has taken BizTalk developer training, and optionally, BizTalk administrator training
5,	BizTalk application and host usage defined	Are standard hosts used, or are additional ones defined? Does this project belong in an existing BizTalk Application, or a new one?
6,	Reusable assets are factored into common applications and assemblies	Schemas, web services, business rules
7.	Application security requirements are clearly identified	Authentication and authorization

Migrate from Shared Development to Test
	Task	Comments
1.	Code successfully builds and runs all core use cases
2.	Code review is re-executed if significant deviations discovered during initial code review
3.	Performance test plan is in place	Test expected load + 25%; test LOB adapters to confirm downstream system load acceptance
4.	Repeatable release management from development environment is set up	Binding files created, MSI built, scripts constructed; helper components and web services added to MSI
5.	Exception handling strategy is confirmed	System retries, exception logging, resuming suspended messages
6.	Application is ready for multi-box deployment	All hard-coded file system references considered
7.	List prepared of all encountered application and system exceptions, and resolution strategy for each	Used for testing and eventual administration purposes
8.	Need for new host instances or changes to throttling settings considered
9.	All web services are managed by governance platform [SOA Software]

Migrate from Test to Production
	Task	Comments
1.	Application monitoring requested and configured [Microsoft Operations Manager]	Identify events to monitor and notification recipients
2.	Appointed application administrator has BizTalk Operator rights approved
3.	All web services are managed by governance platform [SOA Software]
4.	Debug statements removed and Event Log trace statements removed

Technorati Tags: BizTalk

March 20, 2008

Gracefully Uploading to SharePoint 2007 From BizTalk Server 2006 R1
Scenario: I want to allow BizTalk Server 2006 (R1) to send XML (InfoPath forms) to a MOSS 2007 document library without resorting to hacks.

Resolution: I can’t use the out-of-the-box BizTalk SharePoint adapter (only BizTalk Server 2006 R2 works natively with MOSS 2007) so I decided to utilize the available SharePoint web services to upload my file. I wrote a wrapper web service to (a) encapsulate some additional logic (b) shield the BizTalk developer from understanding SharePoint services and (c) require no usage of the SharePoint 2007 object model.

What did this solution look like? I decided to use the CopyIntoItems method available on the SharePoint Copy web service. This allows you to send a byte array of data to a document library and have it appear as a new document. To hit the WSDL for this service, you’d go to:

http://<your sharepoint base url>/sites/<site name>/_vti_bin/Copy.asmx

Here you’ll see the CopyIntoItems operation. My wrapper service starts with a couple “using” statements …
```
using System.Net;   //for NetworkCredentials object
using System.Text;  //for encoding bytes
using System.IO;    //for stringwriter
using System.Xml;
```
Next I have my wrapper operation …
```
[WebMethod]
public string UploadXmlToSharePoint(string docToUpload,
string siteRoot,
string docLibrary,
string fileName,
string userName,
string password,
string domain)
```
I’m taking the XML document input as a string in order to make the schema easier in BizTalk, and, ensure I don’t lose my InfoPath processing instructions when transporting over the wire (which seemed to be happening when I used an XmlDocument type input parameter). Also note that I’m taking in a user/password/domain combo. This is to allow for reuse later down the line. The account used to call the SharePoint service MUST be a site administrator, so I’m making it an explicit parameter. The first thing I do inside my operation is build up the destination Uri based on the input parameters.
```
//build full destination Url
string destinationPath = siteRoot + "/" + docLibrary + "/" + fileName;
```
Next I have to take the input string and convert it to the byte array required by the MOSS web service …
```
//convert string to byte array
byte[] fileIn = ConvertDocToBytes(docToUpload);
...
private byte[] ConvertDocToBytes(string xmlString)
    {
        ASCIIEncoding encoding = new ASCIIEncoding();

        return encoding.GetBytes(xmlString);
    }
```
Now I need to instantiate some values needed by the MOSS service. First we have the “result” object which conveys the state of the copy transaction. Then I have a “FieldInformation” array which can be used to pass in specific field values. Note that you CANNOT pass in a null value here, or else you get a cryptic error when calling the service. You can make it blank, but don’t use a null parameter in its place. Finally, I create a destination Uri array.
```
//holds MOSS service response values
SharePointSvc.CopyResult[] results;

//required fieldinformation array
SharePointSvc.FieldInformation fieldInfo =
    new SharePointSvc.FieldInformation();
SharePointSvc.FieldInformation[] fieldInfoArray = { fieldInfo };

//destination url (notice that it's an array, meaning
//multiple sites COULD be targeted
string[] destUri = { destinationPath };
```
Now I can actually call this puppy. After instantiating the web service proxy class (generated by the Add Web Reference command), I need to provide explicit credentials.
```
//create instance of web service proxy
SharePointSvc.Copy copy = new SharePointSvc.Copy();

//pass valid credentials
copy.Credentials = new NetworkCredential(userName, password, domain);

//call primary operation;  sourceUri, doesn't matter here
copy.CopyIntoItems(
     "http://none",
     destUri,
     fieldInfoArray,
     fileIn,
     out results);
```
The last step is to actually check the “result” object for errors and return any errors back to the caller.
```
//check for error and return final result
if (results[0].ErrorMessage != null)
  {
    return "Error: " + results[0].ErrorMessage;
  }
  else
  {
    return "Success";
  }
```
Sweet. After building and deploying this service, I can call it from any client, BizTalk (2004/06) included. I’ll obviously want to securely store my credentials (using Enterprise Single Sign On) and not embed those in my client directly.

So if I call my service, and pass in a plain old XML file, it shows up in my document library as expected.

Now, if I send an InfoPath document to a Forms Library set up with an InfoPath template, the result is this …

Nice! The document is recognized as an InfoPath document (see the icon), and, the promoted columns are properly loaded.

So, if you’re interested in a fairly easy way to programmatically upload documents to a MOSS 2007 library, without having to use the SharePoint object model or BizTalk adapter, this web service might just work for you.

Technorati Tags: SharePoint, BizTalk
March 17, 2008
How Do You Know If You’ve Reached Practical SOA Maturity?
I’d like to think that my company is fairly far along the SOA path, but we’ll be spending time this year looking to refine our SOA reference architecture and map our progress on a SOA maturity model. Most of the maturity models out there have some sort of scale which ranges from “starting with basic services in pilot projects” and graduates to “enterprise services and process orchestration” all the way to “agile, event-driven environments.” If you’re unfamiliar with concept, check out what Sonic has to say.

All that said, I was thinking about some of the lower-level, tangible ways that I could observe whether we were indeed becoming more advanced in our SOA journey. So, I put together a short list of 10 criteria that would indicate to me that we were doing more than paying lip service to the architectural principles of SOA. Let it be noted that I don’t think my company does many of these, but you can’t know what you’re shooting for if you don’t throw up a target.
1. Organizational agreement on software validation processes and procedures. Working in an FDA-regulated environment means that many of our systems require vigorous software validation processes. Now considering that the nirvana of an SOA environment is that business and system processes can be constructed using the farm of existing reusable services, how do you properly validate this new system? That is, must you revalidate each of the reused services that make up the new system in order to validate the entire new system, or do the software validation processes take into account composite applications? The thinking of some is that a key criteria for this is risk. Assess the risk points of the services working together, and then focus the validation effort on only those areas called out in the risk analysis. I tend to like this approach since it’s progressive enough that we aren’t clumsily and coarsely retesting existing services unnecessarily, and instead focusing on what the FDA cares about the most, risk.
2. SOA implementation is not platform-specific. I don’t want our SOA infrastructure to built solely upon Microsoft technology. I don’t want it solely built with Java technologies either. The services that we expose should all be supportive of cross-platform standards (and data types) and be consumable by any of the (many) technology platforms we have in place. I should be able to call, with no issue, any exposed service from either a Windows, Unix, Linux environment and/or Microsoft or Java client. This touches on the next point.
3. Service testing criteria should be defined and rigorous. We shouldn’t simply be unit testing our service with blinders on that limit our scope to our target system. If the goal is to maintain enterprise services, then each service should be challenged with cross platform invocation, and be proven to work as part of a composite service or process. It’d be great for us to have a short, but comprehensive checklist that any service, regardless of situation, is tested against.
4. Common set of enterprise objects is available. We’ve done this with our SAP implementation, where many of the core SAP objects are represented via XSD schemas and it is assumed that if you receive data from SAP for these functional areas (e.g. employee data, vendor information), it will conform to the enterprise schema. Now, it’s impractical that every existing system and data source will be evaluated and a complete set of enterprise objects will be defined. That doesn’t make sense. But, we would probably benefit by having a “customer” object that all our various system services would accommodate. Once you’ve defined these objects, the next critical piece is publishing them for others to discover.
5. Service repository contains services AND metadata. We use SOA Software for our service directory, but simply discovering and listing services isn’t enough. We also need clear guidance around how to best *describe* the service. What is the metadata description needed for someone to not only discover this service, but know if it fits their needs, and how to consume it? The first step is to find and register the services, but maturity will only be reached once we’ve properly cataloged and described them.
6. Service lifecycle is well defined. Specifically, what is the change management, release management, and service ownership story? Does a single group *own* all the services, or is there distributed ownership and management? When changes are needed, how do you ensure that the service is appropriately versioned, and that breaks to the contract do not cause issues for existing consumers?
7. Metrics are captured and actionable. While SOA Software provides us the means to observe all sorts of details on service usage, it’s arguably even more important to define how to act upon this data. Do you defined a BAM dashboard? What happens if specified SLAs are failing? What’s the policy here?
8. Service developers incorporate common service patterns. I’ll specifically call out communication patterns. Can our developers articulate why they would choose a request/response pattern, asynchronous one-way pattern, or publish/subscribe pattern, and know how or when to use callbacks? Many services are well served by an asynchronous processing model, but I wonder how many developers are comfortable with that. Also, how many of our services support MULTIPLE communication patterns? In many cases, it makes lots of sense to support not only a request/response channel, but also a one-way channel.
9. Developers build services that can ACTUALLY be reused. Some folks may say that reuse is critical to an SOA architecture, but plenty of others will say that reuse isn’t a requirement and can actually hamper, not help, your design. However, developers should be considering reuse while modeling their services, even if that only means that they carefully evaluate the service contract and exposed operations. Likewise, can the service you’re building be detached into smaller, more modular components? Maybe the broader application-specific service you’re writing has no possibility of reuse, but potentially a smaller part of the operation does include a repeatable process. Sufficiently considering enterprise reuse is often one of the hardest things to do on a project. How can you guess what someone MIGHT do with your service months from now? But, taking a small slice of time to debate both “who” and “how” your service might be reused may yield design decisions otherwise not considered.
10. New architects/developers can quickly understand our SOA. Can we grab a new college-hire or a seasoned developer and have them build services that comply with our SOA vision? If our SOA principles and design criteria aren’t clearly articulated, we will quickly find an excessive amount of deviations to our master plan. We don’t need a 400 page manifesto, but it is important to call out our standards for envisioning, designing, testing, security, deploying, consuming and managing enterprise services if we’re truly committed to building a vibrant service-oriented infrastructure.
Thoughts? Do you find that you or your company already do many of these? Or do you think any of these criteria are unnecessary or incomplete?

Technorati Tags: SOA
March 13, 2008
BizTalk Database Architecture Poster Now Available

The BizTalk product team keeps pumping out posters, so I guess that I’ll keep highlighting them. This particular one focuses on the core BizTalk databases, their functions, and how they relate to each other. Specifically, you’ll find each BizTalk database alongside it’s role, functional events, data movement events and more. Given that SQL Server databases form the heart and soul of BizTalk Server, it’s fairly important that any self-respecting BizTalk person knows this information.

I can’t imagine that there are many more poster ideas left for the BizTalk team to produce. I mean, what’s left? Security roles? Functoids? Maybe they’ll mail it in and produce a poster of the product team itself playing a spirited game of freeze tag.

Technorati Tags: BizTalk

March 4, 2008
Wally the Architect

After reading today’s Dilbert comic, I’m not sure Wally is the standard-bearer for architects that I want to get behind …

March 4, 2008