Well, thanks to Chris for letting me know that ESB Guidance for BizTalk Server was added to Codeplex.
I’m actually deploying an application this week based on the Exception Management code. I changed it around a bit, but having these bits accelerated my development significantly. Now I need to find a way to upgrade to these current components!
[Series Links: Part I / Part II / Part III / Part IV]
In the first three parts of this series, I’ve shown you how SOA Software enables “last mile” web service management and configuration. Now, let’s focus on HOW you call a service that is managed by SOA Software Service Manager. If you add a policy to a service which requires specific authorization headers, then clearly you expect the service caller to add those headers. However, a developer probably doesn’t want to get bogged down in adding SAML tokens or applying digital signatures.
SOA Software provides a Gateway Service which sits in between the client and service endpoint. A .NET/Java SDK is provided for clients to interact with this Gateway Service. Using the SDK, developers can work with an API that provides support for:
So, by using functions in a rich SDK API, the developer can avoid building complexity and guesswork into the construction of a service message managed by a robust policy. Now, let’s make it even easier. The last option in my list above (“Endpoint auto-configuration”) means that instead of asking the service caller to know how to pack up the service payload, do it for them.
Within the SOA Software Service Manager you can set up a friendly identifier for the web service. Then, from client code, you can use the Gateway SDK to lookup all the policy information for a given service, and build up the message accordingly. That is, the developer writes 1 line of code to apply all the necessary policy bindings. The Gateway service then receives this command (auto-configuration) from the client, and packs up the message with a format required by the policy before forwarding the service call on to the destination. Cool!
Now if I call a SOA Software managed web service (which has an “authentication” component in its policy) from BizTalk using the standard SOAP adapter (with an orchestration feeding it), I get the following error:
Error details: SoapHeaderException: An error has occurred authenticating based on Credentials
Great! So how do we get around this? My first thought was a pipeline component which would call the Gateway SDK code. I tried this, but it failed. The Gateway SDK code needs to be on the same calling thread as the actual service call. So, I needed to move this code as close to the adapter as possible. Thanks to the stud support folks at SOA Software, they suggested doing a SOAP send port with a proxy class (versus using the default “Orchestration web port” settings). So, I auto-generated a proxy class using wsdl.exe, and added the “gateway bridge” code to the corresponding web method.
My send port then looked like this …
I also had to change the send port’s URL to point to the Gateway service URL. So now, no part of my project points to the ACTUAL web service. Rather, I point to the Gateway service which adds all the necessary policy code before forwarding traffic to the real web service endpoint. Making these changes resulted in BizTalk working perfectly. No custom adapters, no need to unnecessary interject orchestration and fairly simple maintenance.
Now, one concern I had was that using this architecture, my service caller (e.g. BizTalk) forwards a web service call to another box hosting the Gateway service, which then forwards the message on to the final service endpoint. Because service policy information is applied by the Gateway service, I can be confident that no one can sniff or tamper with messages leaving the Gateway machine. However, what about that call from my client TO the Gateway? Do I now have to do some HTTPS transmission JUST to get the Gateway?
Thankfully, the SOA Software folks thought of this. A parameter in the API call to the Gateway actually enables the payload to be fully encrypted. How do I test this? I’m using the great tools from PocketSOAP. You could use TCPTrace, but I actually like ProxyTrace more since the BizTalk setup is trivial. Using ProxyTrace, I can see the actual message being sent by the BizTalk SOAP adapter.
After you start ProxyTrace (and tell it which port to listen on), you simply change your SOAP adapter (or HTTP adapter) “Proxy” tab like so:
Once I call my service, I can now see the raw payload sent it, and the raw data returned. As you can see below, my message to the Gateway is in clear text (the password has been automatically hashed), and no funky policy headers have been applied yet.
If I rebuild my SOAP proxy class (which contains the Gateway SDK code) to have the “encryptMessage” set to “true”, then my transmission out of the BizTalk box looks like this:
I love that. Very simple, quite effective. If I check the recorded message in the SOA Software Service Manager portal site, I can see that after the Gateway processes the message, all the required policy elements have been added.
Takeaway: To attach management and policy functionality to a service does NOT require changing anything in the service itself. No changing configuration files, code, etc. To CALL a service that is managed by SOA Software, you can use either the Java or .NET Gateway SDK to abstract the actual complexity required to attach relevant policy data.
Great stuff. These folks are working on cutting-edge things, and I’m constantly surprised at the overall thoughtfulness and completeness of their platform. Highly recommended.
[Series Links: Part I / Part II / Part III / Part IV]
In the last post we looked at how SOA Software is used to manage and maintain web services. Today, let’s look at how we can configure web service “policies” to do everything from encryption to load balancing.
So far we’ve seen that a give service managed by SOA Software can have a “policy” attached to it. What is a “policy” really? The types of policies we’ll look at here include “management” policies, “SLA” policies, and “access” policies. For a BizTalk developer, think of a management policy as a pipeline. The management policy configuration page looks like this:
You have a series of components to handle the request message, and series of components to handle the response message. Again, very much like a BizTalk pipeline. For the Record component in this template, you can configure it as such:
You can snag either the entire payload, or choose to grab only a specific record(s). Odds are, you’d only have a policy do this much recording of data during a test or debugging scenario. But, it’s quite useful if you’ve got a series of complex components (encryption, etc) and want to print out the result after a particular stage.
The Security policy component is quite powerful. You can perform authentication checks to ensure that the service client is a valid user in a defined identity system. After choosing which identity system the policy should confirm against, you next choose the means of capturing the actual identity. Choices include SAML, x509 certificates, HTTP Basic Auth, and more. From this component you can also apply or verify digital signatures.
If you choose to, you can also apply WS-Encryption and WS-Decryption capabilities. The WS-Encryption lets you choose the way to encrypt, and then WHAT to encrypt. So you could identify 4 nodes in your SOAP payload that contain sensitive data, and ONLY encrypt those fields.
You could also choose to manipulate the message a bit by using the Strip Element component. Let’s say that you’ve already verified a signature, or received credentials in the payload that are now unnecessary. You could strip those out before sending the message on to the next hop. Nice.
For performance reasons, you may also want to use a Caching component. In this case, the Policy will compare the SOAP body/envelope and if it matches an instance in cache, the cached response message is immediately returned.
Other valuable management policy components include:
Those were examples of “management” policies in SOA Software. You also have the ability to create additional policy types. One interesting one is the SLA Policy. In this type of policy, you can dictate performance-related metrics that should result in a system level alert. For instance, my policy below says that if SOA Software encounters more than 25 SOAP faults in a 1 day period, raise an alert. And remember, this specific policy can be associated with individual services and operations.
There’s also the concept of Access Policies which can be used to restrict traffic to a given service. If you had an external vendor calling your service, you could create a policy that says that a particular user/company can only execute the service 500 times a day, or, only between the hours of 8am-5pm Monday through Wednesday.
Now you could have the concern that managing these policies, and who is using them, could be a nightmare. What sort of dependency tracking do you get? The answer is, surprisingly strong reference management. In my Policy Overview screen, I can see how many service operations use a given policy.
I can then drill even further, and see each operation using a given policy. That alone is useful, but what if a mass update needed to happen? From this view, I can make bulk changes to each service that uses the policy!
That’s huge. Otherwise, even 50 services could be a challenge to maintain, not to mention 500!
It’s a fairly safe assumption that the “policy” functionality of SOA Software is where you find significant business value. All of these policies are applied, and enforced, without touching your service, or creating code. I suspect that my company will use the security-related components most frequently. The non-security policy components will definitely find a home in various policies, but we’ll see a universal adoption of security components off the bat.
As you’ve seen so far, managing existing services is easy. The only place where a developer needs to be cognizant of the fact that a service is managed is when they CALL that service. The next post is where I bring it all home. Specifically, how can BizTalk Server (or ANY .NET/Java application) make accurate calls to a service managed by SOA Software?
[Series Links: Part I / Part II / Part III / Part IV]
In the previous post, we did a high level look at SOA Software and the core modules of that platform that my company is using. Now, let’s dig a bit deeper in one of the base subsystems, the Management Server.
At its core, the SOA Software Management Server allows you to configure your services by applying appropriate policies. The Management Server also provides the monitoring component, but we’ll discuss that in the “monitoring and alerting” post that will follow this one. Here we’ll focus a bit more on registering and managing services using SOA Software.
To manually register a service, you can start from the SOA Software Service Manager Dashboard. The wizard window asks you for the WSDL of your physical service.
Then you can choose which operations on the service you wish to manage. “Managing” the operation gives you the ability apply policies, record and monitor data, and so forth.
Finally, you choose a default policy to use. This can be changed after the wizard is complete.
Now, SOA Software has the concept of a “virtual service”. These virtual services act as proxies that can be deployed on different machines, and, may be an aggregate of operations from multiple “physical” services. For instance, I may create four different web services that interact with a “vendor” object. Maybe a few operations on two of those services are going to be publicly exposed to outside parties. I can create a single “virtual service” made up of operations from two of those services, and deploy that “virtual service” to a locked down box in the DMZ. The same “physical service” may be part of multiple “virtual services”. Now because you can apply different policies to each operation in a service, it’s conceivable that you attach different policies to the same “physical service” (by virtualizing it) in order to target different audiences. Maybe the “GetVendor()” operation, when used internally, goes through a particular virtual service that only confirms that the user has a valid Active Directory account. However, when “GetVendor()” is exposed publicly, THAT virtual service operation may require a policy that does encryption/decryption AND authentication. Or, you may have a different service level agreement (SLA) for external parties than for those internal users. No problem, just apply different SLA policies to each individual virtual service. When I create a virtual service, I can choose which operations, from which services, to include.
Note that the physical services being aggregated could be on different servers, and even different application server platforms. No reason I can’t have a virtual service that combines related operations from services in IIS 6.0 and Weblogic.
Services may also get “auto discovered” if they are deployed to a box with the SOA Software agent running. Once a service is managed (either through auto-discovery, or manually), you open up a wealth of management options for that particular service. The most eye-catching part is the real-time feed of usage data.
This Flash object updates in real time so that you can see each service call and how long it takes. By mousing over the bars, you can see specific details about the time the operation took to execute. Nice!
You may think that with all these “virtual services” floating around that it could be a challenge to remember what the correlations are between services. Have no fear, the “Service Relationships” widget provides another Flash-based view of the connection between virtual and physical services.
By moving your mouse over the “services” in this window, you can see the details about each service in the relationship tree.
Finally, the Portal provides plenty of ways to see metadata about services and perform maintenance tasks. The “Actions” widget associated with each service exposes various functions.
You can view a copy of the WSDL, virtualize a given service, and much more. One of my favorites is the “Test Service” function. This actually allows you to execute the service itself. My initial thought was that this would be similar to the ASP.NET Web Services testing page that is shown in IIS. That is, if you have simple type parameters (ints, strings, etc), then test away. However, the default ASP.NET Web Services test page doesn’t allow you to test complex type parameters. HOWEVER, the SOA Software web services tester actually parses out the complex type and lets you test those as well!
That’s great stuff. Unexpected, and very useful.
The SOA Software Management Server is a great way to centralize management and ownership of services. The ability to generate proxy services that abstract the underlying services has enormous implications for distributed environments. In the next part of this series, I’ll tackle the Policy Manager subsystem.
[Series Links: Part I / Part II / Part III / Part IV]
Recently, the company I work for purchased a web services management platform solution from SOA Software. This post starts a short series where I will highlight the key features of the platform, and, how to integrate BizTalk with this solution.
SOA Software provides us with that “last mile” web services capability set that should help my organization realize our vision of a truly enterprise SOA. Specifically, SOA Software provides us with four key functions:
In this first post, I’ll do a very high level overview of each of those functions.
Centralized management, including automatic service discovery
Each of our web servers will have a SOA Software “discovery” agent installed which detects when new web services are deployed to the box. The supported application servers include Websphere, Weblogic, Tomcat and IIS. So this plays very well in a Microsoft, or non-Microsoft shop. Once a web service has been discovered, it can be centrally managed. “Managing” a service can mean applying security policies, establishing service level agreements, apply access/deny rules, and more.
Fully UDDI 2.0 compliant service registry (UPDATE: UDDI 3.0 compliance as well)
It’s nice that SOA Software provides us a service registry out of the box. No need to purchase and integrate another company’s registry solution. The registry has a fairly rich “search” capability where I can search for service by name, usage, category and more.
What are categories? Once a service is managed, you can group the service using user-defined categories. Much nicer than having a registry that just shows a dump of every service with no way to organize them.
Application of service “policy” via configuration
While you could potentially do without the previous two features (even though it makes life significantly easier), the SOA Software Policy Manager service is where you really get your money’s worth. Without TOUCHING the code or configuration files of a deployed service, you can apply a “policy” to that service. Because each service is hosted on a web server with a SOA Software management agent, all service requests can be intercepted. SOA Software acts as an intermediary between the service client and service provider. A service policy may include:
Those are examples of “management” policies. I’ll also show you “SLA” policies, and more in the later post. Each method of a given service can have a different policy attached. I can’t stress how cool this stuff is. When someone builds a service, they don’t have to do anything special for this service management capability. All of it occurs at the management point, not development point.
Full-featured monitoring and customized alerting
A robust monitoring module is so important if you want to evaluate service usage trends in your organization. I can capture the payload of the SOAP messages coming in and out of the services. This includes the ability to see the services before/after policies have been applied. The view below shows my message after a SAML token has been applied.
I can also see various reports about service response time, usage and more. So far, that is just passive reporting. I have to go query for data to discover a problem. What if I have a SLA set up, and NEED to be very proactive about service interruptions or problems? You can subscribe to alerts already configured, or, create your own. I can build an SLA Policy to ensure that if response time goes above a certain amount, or the number of SOAP faults exceeds an acceptable amount, then send me an email message.
In the next few posts, I’ll show more details about these key subsystems. Finally, I’ll show how BizTalk can call one of these SOA Software-managed services. The service *caller* requires a bit of code (in order to inject the headers necessary to conform to the policies), but SOA Software has made it fairly straightforward.
One of the technology backbones of my company moving forward is the use of BizTalk to “fan out” messages from our ERP system (SAP). Instead of having hundreds of direct interfaces to SAP by downstream systems who need real-time data, we can use BizTalk to receive a single message from SAP (XI), and distribute it to all interested parties. One issue brought up recently was “ordered delivery” and ensuring that if a downstream system needs messages delivered in the order sent by SAP, that BizTalk honor it. So, I set out to test a variety of scenarios using BizTalk’s built-in ordered delivery capabilities.
Setup: I exposed a BizTalk schema as a web service to simulate our publishing receiver from SAP. A simple Windows Form is sending messages to this web service, and stamping each one with a sequence number. I then built a couple of web services to act as “subscribers” of these SAP data events. These web services publish the data to a database table, thus letting me see the exact order of messages delivered. Each SOAP send port has a subscription based on message type.
Scenario #1: Send port, no ordered delivery
When a series of messages are published to the receiving web service and routed to the subscriber without ordered delivery, the result may look like this:
As you can see, even on my single development machine, the order gets mixed up. This is due to batch processing and the multi-threaded nature of BizTalk message distribution.
Scenario #2: Send port, ordered delivery turned on
To turn ordered delivery on, all we need to do is check a box on the send port (this assumes that your inbound transport can deliver messages in order. Examples include MSMQ, SOAP and HTTP).
The result in the database looks like this:
So you can see that all the messages are in sequential order.
Scenario #3: One send port no ordered delivery, one send port WITH ordered delivery
I wanted to prove that ordered delivery only impacted the corresponding port. When sending the message into BizTalk with two send ports subscribing, the data looked like this:
“Subscriber #1” kept everything in order, but “Subscriber #2” delivered messages a bit more haphazardly.
Scenario #4: Non-ordered delivery send port, no retries, and error in service
So what if the service raises an error? When there are no retries, and the send port doesn’t have an ordered delivery requirement, the result looks like this:
The bad message (#12) is simply skipped (because it is now suspended).
Scenario #5: Ordered delivery send port, retries enabled, and error in service
How about for an ordered delivery send port? What if there’s an error in the service call, and retries are turned on for the port? Also, the “cancel if error” is turned off. Check this out:
The messages queue up waiting for the first one to retry, and hopefully succeed. We’ll see more of this in a moment.
Scenario #6: Ordered delivery send port, NO retries, and error in service
This time, there’s an error in the service, and no retries. Also, the “cancel if error” flag is still turned off. The result is:
So the messages still get delivered AFTER the bad message has been encountered. That’s expected, since we told BizTalk to keep going, even for the ordered delivery port.
Scenario #7: Ordered delivery send port, NO retries, “cancel if error” turned on, and error in service
What if we reproduce the previous scenario, BUT, turn “cancel if error” on? This flag can be set on the send port here:
What you’re telling BizTalk is that if any message fails for this ordered delivery port, stop processing until this error can be corrected. This is useful if you’re concerned that an “insert new contact” message failed, but you expect a “modify existing customer” to be following. Clearly the “modify” message will fail unless the “insert” gets figured out. The result of this?
Processing stopped after message #11 because message #12 failed. The send port is still “started”, but you’ll find a suspended message. If you open it up, you’ll see that all following messages are queued up until the offender gets resolved.
Scenario #8: Send port stopped, and restarted for a non-ordered delivery send port
What if we KNOW that a downstream system is unavailable (e.g. maintenance) and want to prevent the inevitable failure of delivery? Maybe we want to shut off the send port, queue up the messages, and once the unavailable system is back online, open the distribution pipeline again. For a non-ordered delivery port, after restarting a stopped send port, the output looked like this:
The messages are in a crazy order, as you can see.
Scenario #9: Send port stopped, and restarted for an ordered delivery send port
Final scenario. What if we stop an ordered delivery port, and then start it back up later? Do we retain the correct ordering? When I sent five messages in, I got a suspended instance with five messages:
If I send in three MORE messages, they actually get added to this suspended instance:
Once I finally turn the subscriber service back on, all messages were sent in the same order received.
So there you go. Ordered delivery is a fairly powerful concept, and not particularly hard to do with BizTalk. It greatly impacts performance because the send port won’t send subsequent messages until a delivery confirmation is received, but, in many cases that performance impact is outweighed by business requirements.
Technorati Tags: BizTalk
Quick follow up to today’s post on production-ready BAM tactics.
If you have folks accessing the BAM Portal, make sure they’ve installed the necessary components. Specifically, if they want to view aggregation (pivot tables), they must install:
You can get both of those download from here. Thanks to Paul Wu’s blog for the tip.
Finally, it’s a BAM sort of day, so check out Darren’s great post on BAM best practices.
Technorati Tags: BizTalk
I recently went through the process of deploying a BAM model to our new BizTalk infrastructure, and learned a few things about BAM security and deployment along the way.
Given that most BizTalk architects/developers probably play with BAM on a single fully installed machine (BizTalk, SQL, IIS, etc) while running with highest-level permissions, sometimes certain steps can be taken for granted.
To start with, my production environment contains clustered SQL Server 2005 servers and a specific SQL instance created for the BizTalk databases. Both SQL Server Analysis Services and SQL Server Integration Services are installed in the cluster as well. If you have this sort of environment, you’ll need to modify SSIS before continuing. Specifically, you must change the MsDtsSrvr.ini.xml file so that the SSIS points to a named instance of SQL Server (see the Microsoft documentation for details). If you do NOT do this, then running the bm.exe BAM tool will result in everything LOOKING fine, but in fact, no SSIS/DTS packages get created anywhere.
Now, to run the bm.exe, which builds up all the BAM infrastructure (tables, procedures, triggers, packages, cubes, etc), you have to have SSIS on the same machine as the tool itself. Got that? So you can’t run this from a standalone BizTalk box and expect it to work. Another option (instead of installing SQL tools on the BizTalk box) is to install the BAM tools alone on the SQL box. I’ve reviewed a few things, and am fairly sure this is the case, but if anyone wants to challenge that assertion, feel free.
Let’s talk about security. Specifically, adding users to views. Again, most times when you’re developing BAM solutions, you take for granted that you can go to the BAM Portal and views magically appear. But when you’re not running as an Administrator, those views aren’t visible by default. What I did (as practice before doing this in production), was create a local group in my development environment. Then, I ran the following command:
This command does a few things. First, it adds that user/group to the BAMPrimaryImport table as a “user.”
This allows anyone in that group to look at data in the BAM view. If your BAM model contains aggregations, then SQL Analysis cubes get created by the bm.exe tool. A new “role” gets created for you in SQL Analysis services as well …
Now by default, this group is empty. But when you run the “add-account” command, the designated user/group ALSO gets added here.
Nice! So instead of doing “add-account” for every individual user, you should require a group to be set up for a particular BAM deployment. If a user tries to view the BAM Portal and is NOT in the designated group, they’d see this …
As soon as I add the logged on user to the pre-defined Windows group (with explicit BAM access), the same user sees this …
Cool stuff that I haven’t found demonstrated much. I personally spent hours trying to find out why my freakin’ SSIS jobs wouldn’t get created, so the whole “change the obscure SSIS configuration file” might save someone time later on. Any other BAM deployment points folks want to add?
Technorati Tags: BizTalk
Lots of stuff today about BizTalk Services, the new offering from Microsoft that exposes an “Internet Service Bus” built upon BizTalk. Good summaries by Mick and Chris. There’s also a very nice article in eWeek
(not the one linked to from the Labs site) entitled Microsoft’s BizTalk Services Simplify SOA that talks about the strategic important of these software services.
While intellectually interesting to me, I’m hard-pressed at the moment to see a use case within my organization itself. Steve Martin of the BizTalk team has a good write up and hits the target audience: We see BizTalk Services as a complement to “traditional” BizTalk Server uses on premise. As you need to coordinate SOA on a broader scale beyond the organization, we see the introduction of hosted services as one way to help support federation of business process, messaging, and identity across boundaries.
Well said.
Technorati Tags: BizTalk
So there’s a new website/newsletter for BizTalk addicts created by some great Microsoft Technology Specialists. Go to BizTalk Hotrod and check out the first issue. The newsletter is surprisingly well put together, and was actually a quite educational read.
I must admit, though, when I got an email saying “Check out BizTalkHotRod.com”, I was definitely expecting some unsavory content. Actually, it sounds like something an over-the-top BizTalk consultant would put on their personalized license plate. I’m not sure I could be friends with that person.
Technorati Tags: BizTalk
Richard Seroter's Architecture Musings 