Richard Seroter's Architecture Musings

Author: Richard Seroter

  • Production-Ready BAM Security and Deployment

    I recently went through the process of deploying a BAM model to our new BizTalk infrastructure, and learned a few things about BAM security and deployment along the way.

    Given that most BizTalk architects/developers probably play with BAM on a single fully installed machine (BizTalk, SQL, IIS, etc) while running with highest-level permissions, sometimes certain steps can be taken for granted.

    To start with, my production environment contains clustered SQL Server 2005 servers and a specific SQL instance created for the BizTalk databases. Both SQL Server Analysis Services and SQL Server Integration Services are installed in the cluster as well. If you have this sort of environment, you’ll need to modify SSIS before continuing. Specifically, you must change the MsDtsSrvr.ini.xml file so that the SSIS points to a named instance of SQL Server (see the Microsoft documentation for details). If you do NOT do this, then running the bm.exe BAM tool will result in everything LOOKING fine, but in fact, no SSIS/DTS packages get created anywhere.

    Now, to run the bm.exe, which builds up all the BAM infrastructure (tables, procedures, triggers, packages, cubes, etc), you have to have SSIS on the same machine as the tool itself. Got that? So you can’t run this from a standalone BizTalk box and expect it to work. Another option (instead of installing SQL tools on the BizTalk box) is to install the BAM tools alone on the SQL box. I’ve reviewed a few things, and am fairly sure this is the case, but if anyone wants to challenge that assertion, feel free.

    Let’s talk about security. Specifically, adding users to views. Again, most times when you’re developing BAM solutions, you take for granted that you can go to the BAM Portal and views magically appear. But when you’re not running as an Administrator, those views aren’t visible by default. What I did (as practice before doing this in production), was create a local group in my development environment. Then, I ran the following command:

    bm.exe add-account -AccountName:”machine\ProjectBAMUsers” -View:viewname

    This command does a few things. First, it adds that user/group to the BAMPrimaryImport table as a “user.”

    This allows anyone in that group to look at data in the BAM view. If your BAM model contains aggregations, then SQL Analysis cubes get created by the bm.exe tool. A new “role” gets created for you in SQL Analysis services as well …

    Now by default, this group is empty. But when you run the “add-account” command, the designated user/group ALSO gets added here.

    Nice! So instead of doing “add-account” for every individual user, you should require a group to be set up for a particular BAM deployment. If a user tries to view the BAM Portal and is NOT in the designated group, they’d see this …

    As soon as I add the logged on user to the pre-defined Windows group (with explicit BAM access), the same user sees this …

    Cool stuff that I haven’t found demonstrated much. I personally spent hours trying to find out why my freakin’ SSIS jobs wouldn’t get created, so the whole “change the obscure SSIS configuration file” might save someone time later on. Any other BAM deployment points folks want to add?

    Technorati Tags:

  • BizTalk-Based Internet Service Bus

    Lots of stuff today about BizTalk Services, the new offering from Microsoft that exposes an “Internet Service Bus” built upon BizTalk. Good summaries by Mick and Chris. There’s also a very nice article in eWeek
    (not the one linked to from the Labs site) entitled Microsoft’s BizTalk Services Simplify SOA that talks about the strategic important of these software services.

    While intellectually interesting to me, I’m hard-pressed at the moment to see a use case within my organization itself. Steve Martin of the BizTalk team has a good write up and hits the target audience: We see BizTalk Services as a complement to “traditional” BizTalk Server uses on premise. As you need to coordinate SOA on a broader scale beyond the organization, we see the introduction of hosted services as one way to help support federation of business process, messaging, and identity across boundaries.

    Well said.

    Technorati Tags:

  • New BizTalk Magazine (BizTalkHotRod.com) Available

    So there’s a new website/newsletter for BizTalk addicts created by some great Microsoft Technology Specialists. Go to BizTalk Hotrod and check out the first issue. The newsletter is surprisingly well put together, and was actually a quite educational read.

    I must admit, though, when I got an email saying “Check out BizTalkHotRod.com”, I was definitely expecting some unsavory content. Actually, it sounds like something an over-the-top BizTalk consultant would put on their personalized license plate. I’m not sure I could be friends with that person.

    Technorati Tags:

  • Behavior of BizTalk Map’s Scripting Functoid

    I’ve been spending way too much time building BizTalk maps recently, but, I did uncover some interesting behavior of the Scripting functoid that I thought was worth sharing.

    My opinion is that using “Inline C#” (or VB.NET or JScript) in the Scripting functoid should only be for very simple operations, not page-long functions. In my case, I was receiving a datetime value in “dd/MM/yyyy” format, and needed to convert it to the XSD datetime format of “yyyy-MM-dd”. There were a few dates I needed to do this for, so I used the Scripting functoid and wrote a simple function like this:

    System.DateTime dt = System.DateTime.ParseExact(param1, “dd/MM/yyyy”, new System.Globalization.DateTimeFormatInfo());
    return dt.ToString(“yyyy-MM-dd”);

    So I simply cut and pasted the same “function” into multiple Scripting functoids. In essence, the map looked like this:

    I wasn’t completely sure what would happen when the XSLT was generated for the map. Would it allow multiple functions with the same signature? Would I have to create functions called “DateConvert1”, “DateConvert2”, etc? When I did this as a test using a trial map and function, even though I had two functoids storing a function called “DoSomethingCool”, the output XSLT looked like this:

    Interesting. So, the XSLT generation engine noticed two methods with the same function name and only put one of them into the map’s XSLT. Lots of questions then.

    • So what happened if I changed the implementation of one of those “DoSomethingCool” methods? That is, same signature, but different behavior inside the method itself. Answer: same result. It looked at the first implementation and generated code for only that one.
    • What if I changed the parameter name on the second Scripting functoid? Answer: same result. Still only one function written out.
    • What if I changed the return type to “boolean” on the second functoid? Answer: same result.
    • Finally, what if I changed the parameter type from “string” to “boolean”? Answer: same result.

    Let’s look at more complete picture of the generated XSLT, containing not just the generated code, but the function call as well.

    Notice that the return type appears irrelevant as it’s simply being printed out in the XML node. Parameter name of “DoSomethingCool” doesn’t matter when the function is called. The input parameter is always converted to “string”, so input type doesn’t matter either.

    So, just be aware of these sort of side-effects of reusing the same inline code function within a map. I like that I can reuse a simple function over and over, but it’s good to know how the code generation actually works. Odds are you’d use an external component for some of these situations, but what’s the fun in that?

    Technorati Tags:

  • Great BizTalk Article In This Month’s MSDN Magazine

    Just got the new MSDN Magazine in the mail, and now the issue is online. Check out a great read on 8 Tips And Tricks For Better BizTalk Programming co-written by my old pal Marty. Fairly useful stuff.

    Technorati Tags:

  • Initializing BizTalk Correlation Sets In a Loop (Thanks Tomas!)

    You ever find that newsgroup post on a technical subject that absolutely saves the day? One of my projects has an orchestration that calls a web service. I call this service in a “loop” because if a call fails (for “business” or “technical” reasons), it may need to retried. So, until I set a “ProcessComplete” flag, the loop continues.

    That said, if I DO get an error, I send the message out for review. If I don’t get a response within an hour, I auto-retry the service again. This is done with a transactional scope (long-running) and a timeout. So naturally I have to use correlation to send a message out of my orchestration, and potentially receive a message back. However, after setting this up the first time, I got a
    Correlation set may be initialized only once
    error. This makes sense since I had a global correlation set, and was potentially initializing it over and over in a loop. After staring at my computer for 4 minutes thinking, I did a quick search online and came across a newsgroup post where my new best friend, Tomas Restrepo, reminded me that a *scope* can have its own local correlations sets.

    Sure enough, after moving the correlation set to the local scope, everything compiled fine.

    Technorati Tags:

  • Calling .NET Components From Inline XSLT In BizTalk

    Here’s one for you. Last week I encountered a particularly tricky multi-part mapping scenario. I had to build a destination message that contained groupings from the two source messages. Each record in the first source message created a destination node, and each record in the second source message created a destination node directly beneath the related first source record. To make matters tougher, every destination record has an attribute containing a sequential number. So out of this …

    <source1>
      <Node1><Node1>
      <Node2></Node2>
    </source1>
    <source2>
      <NodeRelatedToNode1></NodeRelatedToNode1>
      <NodeRelatedToNode1></NodeRelatedToNode1>
      <NodeRelatedToNode2></NodeRelatedToNode2>
    </source2>

    The destination was supposed to look like this …

    <destination>
      <Node1 page=”1″><Node1>
      <NodeRelatedToNode1 page=”2″></NodeRelatedToNode1>
      <NodeRelatedToNode1 page=”3″></NodeRelatedToNode1>
      <Node2 page=”4″></Node2>
      <NodeRelatedToNode2 page=”5″></NodeRelatedToNode2>
    </destination>

    The grouping part wasn’t too tough, just used a Scripting functoid with the XSLT Call Template and a little hand written XSL. The hard part was creating the sequential “page” numbers. Those familiar with XSLT know that the “variables” in XSLT are basically constants, so you can’t create a variable and increment it. I considered building some sort of recursion to get my incremented number, but in the end, decided to call a custom .NET component from my map’s XSLT. I built a C# component that had a member variable, and a method called “GetNext()” which incremented and then returned the next sequential number. I then set my map’s Custom Extension XML to an XML document referencing my custom component. Now in my XSLT Call Template I could get the next “page” number each time I built a destination node. Neat!
    See here for an example of doing this.

    Here’s where a “quirk” was introduced. When I deployed this map, and ran multiple documents through it, the first document had it’s paging correct (e.g. pages 1-5), but the next messages had the wrong values (e.g. 6-10, 11-16, etc). What was happening was that somehow this custom C# component was being shared! The “increment” kept counting on each orchestration call! My C# component wasn’t built as a “static” object, and I assumed that the scope of each custom object was the individual map (or orchestration) instance.

    I still have no idea why this happened, but to ensure it wouldn’t keep happening, I added a method to the custom component called “Reset()” which set the counter to 0. Then at the top of the map I call out to that method to ensure that each map starts its counter at 0.

    Thoughts as to why this happens? Wild stuff.

    Technorati Tags:

  • Consuming Web Services in BizTalk Without Web Reference

    Twice last week, I received WSDLs to consume that BizTalk didn’t like. That is, when I would try and load the WSDL via Add Web Reference, I’d get an error like the following:

    Could not generate BizTalk files. Unable to import WebService/Schema. Unable to import
    binding ‘WebServiceBinding’ from namespace
    ‘http://xmlns.company.com/interface/project/SubmitSubjectImpl&#8217;. Unable to import operation
    ‘SubmitSubject’. The element ‘http://xmlns.company.com/project/subject:subjectrequest&#8217; is
    missing.

    I also tried from a “regular” .NET project, and no luck. One of the WSDLs used “imports” (a no-no for importing the web reference), and I’m convinced that the other WSDL’s problem was indeed namespace based. Now in both cases, I didn’t have the luxury of telling the WSDL owner to make changes. So how to get around this? Inspired a bit by Jon’s blog, I decided to try using the .NET Framework tool wsdl.exe. For the WSDL using “imports”, I executed WSDL.exe passing in the WSDL, and parameters for each schema used. Sure enough, I was able to generate a valid C# proxy class. The other offending WSDL was also successfully processed using WSDL.exe. As a result, I now had a proxy class needed to consume this service.

    Next I added that proxy class to a .NET class library, built, and GAC’ed it. Next I used the provided schema files to create messages in a BizTalk orchestration corresponding to the request and response message. Finally, I connected this to a “two way” port in the orchestration.

    After deploying the BizTalk project, I had to configure the necessary SOAP send port. Unlike a “regular” SOAP send port, I was not using an orchestration port as the proxy. Instead, I am now pointing to the proxy class generated by WSDL.exe. On the Web Service tab of the SOAP port, I chose the assembly and method I needed.

    Now assuming everything was set up right, I should have been able to call the web service with no problem. But, if we stick with the base assumption that I’m a total meatball, of COURSE it didn’t work the first time. Instead I got this …


    A message sent to adapter “SOAP” on send port “Company.Dept.Project.SendTibcoSvc.SOAP” with URI “http://machinename:1234/ProcessDefinitions/WebService&#8221; is suspended.
    Error details: Failed to serialize the message part “subjectrequest” into the type “subjectrequest” using namespace “http://xmlns.company.com/project/subject&#8221;. Please ensure that the message part stream is created properly.

    Obviously the proxy couldn’t serialize my XML message. So, I output the message I was sending to the port, and tried validating it against my schema. Sure enough, I got validation errors like The string ” is not a valid XsdDateTime value. and The string ” is not a valid Integer value.. So, I fixed the source problems, and validation succeeded. Finally, when calling the service again, everything worked perfectly.

    Moral of story: you can consume web services WITHOUT using a “web reference” as long as you can build the proxy class. Good stuff.

    Technorati Tags: ,

  • Script for Automatically Creating BizTalk Solution Structure

    Every organization (and developer) has their own development project structure. I’ve set up a BizTalk project structure that my company will use for all new BizTalk projects. To make life easy, I also built a simple VBScript file to automatically build the structure for us.

    My folder structure looks like this:

    I’ve got a spot for deployment packages (Installation), test files, instance data, file adapter pickup/dropoff (Filedrops), strong name key(s), and the actual projects. What’s nice is that each BizTalk project’s strong name key reference looks the same (..\..\..\..\Keys\MyCompany.snk). No hard coded file path. So far, this has worked really well for me.

    To make construction of this structure easier, I built the following script. Just copy and paste it into a .vbs file and you’re good to go.

    ‘*******************************************
    ‘
    ‘ BizTalk Project Structure Creation Script
    ‘ Created by: Richard Seroter
    ‘ https://seroter.wordpress.com
    ‘
    ‘*******************************************

    ‘declare variables
    Dim objFSO, objFolder, strDirectory, strProjectName, strRootFolder, strCurrentFolder

    ‘request data from user
    strDirectory = InputBox(“Set path here (e.g. C:\BizTalk\Projects)”, “Set path”)
    strProjectName = InputBox(“Set project here (e.g. MyCompany.Department.Project):”, “Set project”)

    ‘set root folder of the project
    strRootFolder = strDirectory & “\” & strProjectName

    ‘ Create FileSystemObject
    Set objFSO = CreateObject(“Scripting.FileSystemObject”)

    ‘ Create BizTalk project root folder
    Set objFolder = objFSO.CreateFolder(strRootFolder)

    ‘create BizTalk folder: Documentation
    strCurrentFolder = strRootFolder & “\” & “Documentation”
    Set objFolder = objFSO.CreateFolder(strCurrentFolder)

    ‘create BizTalk folder: Filedrop
    strCurrentFolder = strRootFolder & “\” & “Filedrop”
    Set objFolder = objFSO.CreateFolder(strCurrentFolder)

    ‘create BizTalk folder: Installation
    strCurrentFolder = strRootFolder & “\” & “Installation”
    Set objFolder = objFSO.CreateFolder(strCurrentFolder)

    ‘create BizTalk folder: Installation –> Bindings
    strCurrentFolder = strRootFolder & “\” & “Installation” & “\” & “Bindings”
    Set objFolder = objFSO.CreateFolder(strCurrentFolder)

    ‘create BizTalk folder: Installation –> Packages
    strCurrentFolder = strRootFolder & “\” & “Installation” & “\” & “Packages”
    Set objFolder = objFSO.CreateFolder(strCurrentFolder)

    ‘create BizTalk folder: Installation –> Scripts
    strCurrentFolder = strRootFolder & “\” & “Installation” & “\” & “Scripts”
    Set objFolder = objFSO.CreateFolder(strCurrentFolder)

    ‘create BizTalk folder: Instance Data
    strCurrentFolder = strRootFolder & “\” & “Instance Data”
    Set objFolder = objFSO.CreateFolder(strCurrentFolder)

    ‘create BizTalk folder: Keys
    strCurrentFolder = strRootFolder & “\” & “Keys”
    Set objFolder = objFSO.CreateFolder(strCurrentFolder)

    ‘create BizTalk folder: Projects
    strCurrentFolder = strRootFolder & “\” & “Projects”
    Set objFolder = objFSO.CreateFolder(strCurrentFolder)

    ‘create BizTalk folder: Templates
    strCurrentFolder = strRootFolder & “\” & “Templates”
    Set objFolder = objFSO.CreateFolder(strCurrentFolder)

    ‘create BizTalk folder: Tests
    strCurrentFolder = strRootFolder & “\” & “Tests”
    Set objFolder = objFSO.CreateFolder(strCurrentFolder)

    WScript.Echo “Finished creating ” & strProjectName
    WScript.Quit

    Any changes you might suggest?

    Technorati Tags:

  • Reusing BizTalk Oracle Adapter DSN

    I finally had the excuse to use BizTalk Server 2006 Single Sign On for application configuration data. In one of my current BizTalk projects, I’m using a long running transaction which loops through and starts up other orchestrations. When those orchestrations finish, I’m using partner bound ports to return a response. As I receive batches of responses back, I’m logging the “last update time” to an Oracle table.

    Side note: I’m a believer in *not* using the database adapters (i.e. SQL Server, Oracle) for every possible database interaction. That is, for scalar queries, or simple updates, I like just using a basic utility component and ADO.NET. It’s a little less overhead and saves me from unnecessarily creating orchestration messages. Now, there’s plenty of value in using database adapters (scalability, retries, generated schemas, etc), but for quick and dirty database communication, I’ll stick with the no-adapter solution.

    So, since this solution already includes Oracle database access (and thus has a System DSN set up), I wanted to try and reuse that configuration in my Utility component. The ADO.NET OracleConnection didn’t like the “DSN” connection property, so I used the OdbcConnection instead. However, I still have to enter a password for the connection string. So using the various resources for doing “SSO as a configuration store” (see here, here, and here) I’m able to store the password in SSO, and still reuse my existing DSN configuration. My final code looked like this …


    //get the encrypted password from BizTalk SSO using SSOConfigHelper from BizTalk sample
    string oraclePasswordString = SSOConfigHelper.Read(“ProjCode”, “passwordString”);

    //create the connection string
    oracleConnString = “DSN=ProjDevDb;pwd=” + oraclePasswordString + “;”;

    //instantiate the connection and command
    OdbcConnection oracleConn = new OdbcConnection(oracleConnString);
    OdbcCommand oracleComm = new OdbcCommand(oracleQueryString, oracleConn);

    Now I haven’t implemented any sort of caching, but the “oraclePasswordString” is set in the class constructor, and then I have a method that uses it for the logging function. So, I only make the call to SSO once per process.

    So there you go. Reusing an existing Oracle DSN in custom code, while securely storing the password in Enterprise Single Sign On.

    Technorati Tags: