Richard Seroter's Architecture Musings

Category: SOA

  • "SOA Patterns with BizTalk Server 2009" Released

    This morning my publisher turned a few knobs and pressed a complex series of buttons and officially released my first book, SOA Patterns with BizTalk Server 2009.  It’s available for purchase right now from Packt’s web site and should cascade down to other booksellers like Amazon.com within a week.

    You can find the complete table of contents here, as well as a free, partial sample chapter on the new WCF SQL Server adapter.  What looks like a full, PDF version of that sample chapter (as well as the book’s intro and acknowledgements) can be found here.

    I had three general goals when I started this process almost a year ago:

    • Cover topics and angles on BizTalk Server that had not been broadly discussed before
    • Write the book in a conversational tone that is more like my blog and less like an instruction manual
    • Build all examples using real-life scenarios and artifacts and avoid the ubiquitous “Project1”, “TestSchema2” stuff.

    In the end, I think I accomplished all three. 

    First, I included a few things that I’ve never seen done before, such as WCF duplex binding using the out-of-the-box BizTalk adapters, quasi complex event processing with BizTalk, detailed ESB Guidance 2.0 walkthroughs, and the general application of SOA principles to all aspects of BizTalk solutions.  Hopefully you’ll find dozens of items that are completely new to you.

    Secondly, I don’t truly enjoy technical books that just tell me to “click here, click there, copy this code” so that by the end of the chapter, I have no idea what I just accomplished.  Instead, I tried to follow my blog format where I address a topic, show the key steps and reveal the completed solution.  I provide all the code samples anyway, so if you need to dig into every single detail, you can find it.

    Finally, I decided up front to use actual business use cases for the samples in each chapter of the book.  It just doesn’t take THAT much effort, and I hope that it makes the concepts more real than if I had shown a bunch of schemas with elements called “Field1” and “Field42.”

    So there you go.  It was lots of fun to write this, and I hope a few of you pick up the book and enjoy reading it.   My tech reviewers did a great job keeping me honest, so you shouldn’t find too many glaring conceptual flaws or misplaced expletives.   If you do have any feedback on it, don’t hesitate to drop me a line. 

    UPDATE: The book is now available in the US on Amazon.com and is steadily creeping up in sales rank. Thanks everyone!

    Technorati Tags: ,

  • What Technologies Makes Up an SOA?

    My boss’s boss asked if our architecture team could put together a list of what technologies and concepts typically comprise a service oriented architecture, so I compiled a collection of items and organized them by category.  I then evaluated whether we had such a technology in house, and if so, did we actively use it.  While you don’t care about those latter criteria, I thought I’d share (and ask your opinion of) my technology list.

    Is this fairly complete?  Is anything missing or mischaracterized?

    Category Technology Description Toolset Examples
    Standards XML Markup language for defining the encoding of structured data sets Application platforms, database platforms, COTS packages
    SOAP Protocol specification for exchanging XML data over networks Application platforms, COTS packages
    WSDL XML language for describing web service contracts Application platforms, COTS packages that expose WSDL, IDE tools such as XmlSpy for hand-crafting WSDL documents
    WS* Set of web service standards of varying maturity that address message routing, security, attachment encoding, transactions and more. COTS packages such as SAP, application platforms such as Microsoft WCF
    RESTful Services Architectural style with a focus on resource orientation and interacting with the state of that resource through traditional HTTP verbs .NET Framework 3.5+
    Design Service Modeler Graphical tools for modeling SOA solutions WebSphere Business Modeler, Mashup Tools
    Data Enterprise Entity Definitions Computable representations of shared entities that may span multiple source systems XSD, DDL
    Reference Data Architecture Three components made up of: (a) Operation Data Stores for logical entity definitions that act as “real time data warehouse” consisting of both real-time and batch updated data (b) Data marts for data subset analysis, and (c ) Data Warehouse for enterprise data storage and analysis Oracle, Teradata, SQL Server
    Enterprise Information Integration (EII) Virtual integration of disparate data sources Composite
    Data Service Interface Generation Application for generating interfaces (SOAP/batch) on existing data repositories BizTalk Server 09, .NET Framework, Composite
    Enterprise Service Bus Reliable message routing, transformation, queuing and delivery across disparate technologies BizTalk Server 09, Tibco, Sonic
    Application Adapters Connectivity to cross-platform systems via configuration and/or abstract interfaces BizTalk Server 09, BizTalk Adapter Pack (Siebel, SAP, Oracle)
    ETL Application Bulk data extraction, cleansing and migration between repositories.  Frequently needed for consolidating data in the ODS Informatica, SSIS
    Infrastructure Application Platforms Libraries for building SOA solutions .NET Framework, Java EE
    XML Gateway (Accelerators) Focuses on SOA security and performance issues; typically a hardware appliance WebSphere DataPower; Layer 7
    Complex Event Processing Engine Concept where distinct application or system events are correlated with the goal of discovering critical business information in real time Microsoft StreamInsight, Tibco, Oracle CEP
    Single Sign On Single set of authenticated credentials can be converted to system-specific credentials without user intervention Siteminder
    Data Quality Services Used to cleanse data and validate data quality DataFlux
    Load Balancing Hardware or software solution which sits in front of physical web servers and is responsible for distributing load F5; SOA Service Manager
    Web Hosting Platforms Environment to making web services available to consumers Microsoft IIS, Oracle WebLogic
    Process Integration BPM Server Tooling that supports the modeling, simulation, execution, monitoring and optimizations of business processes BizTalk Server 09;  Lombardi
    Business Rules Engine Software system for maintaining business rules outside of compiled code Microsoft BRE,           Oracle
    Orchestration Services Arrangement of services into an executable business process often designed through a graphical model BizTalk Server 09
    Business Activity Monitoring Aggregation of activities, events and data that provides insight into organization and its business processes BizTalk Server 09
    Service Management XML Service Management (XSM) A platform independent system for administering, mediating and monitoring services SOA Service Manager
    Repository Metadata repository for service definitions SOA Service Manager
    UDDI-compliant Registry Interface for runtime infrastructure to discover and bind to service endpoints SOA Service Manager
    Runtime Policies Attach attributes to service (or operation) that dictates how the service request is secured and processed SOA Service Manager
    SLA Enforcement Define acceptable web service availability with customizable rules consisting of alert code, metric to monitor and an interval SOA Service Manager
    Access Control Contracts Limit usage of service based on distinct number of allowable attempts, time of day window, or based on caller identity SOA Service Manager
    Service Lifecycle Management Stage, status, approvals, dependency tracking, audit trail

    SOA Service Manager
    Security Authorization Verify user identity through LDAP integration, SAML tokens, x.509 certificates Active Directory, Sun ONE LDAP, SOA Service Manager
    Authentication User/group/role at service or operation level SOA Service Manager
    Identity Management Central storage of user profile

    Any feedback is appreciated!

    Technorati Tags:

  • My Book "SOA Patterns with BizTalk Server 2009" is Up for Pre-order

    Well look at that.  I just happen to check my publisher’s website today and noticed that they put my book description online and made it available for pre-order.

    The book, entitled SOA Patterns with BizTalk Server 2009, is due out by May 2009 (hopefully sooner).  I’m still in the process of editing the book and responding to technical review comments, but we’re nearing completion of that phase.  I have three fantastic technical reviewers (Charles Young, Zach Bonham, Ewan Fairweather) who have added a ton of great comments that I’m almost done wading through and addressing.

    This publisher (Packt Publishing) specializes in short, targeted technical books that address specific topics.   I tried to be “short” but still went over my suggested page count by nearly double.  Oh well, hopefully that’s better for the readers. 

    The high level chapter summaries are …

    • Chapter 1 – Building BizTalk Server 2009 Solutions – A brief walkthrough of creating a new BizTalk project from scratch
    • Chapter 2 – Windows Communication Foundation Primer – An overview of WCF with examples of developing, hosting, and consuming WCF services
    • Chapter 3 – Using WCF Services in BizTalk Server 2009 – An explanation of the marriage between BizTalk and WCF and how to both expose and consume WCF services within BizTalk Server
    • Chapter 4 – Planning Service Oriented BizTalk Solutions – A discussion of the core aspects of SOA, the types of services one can construct, available message exchange patterns, and how service-oriented principles apply to BizTalk solutions
    • Chapter 5 – Schema and Endpoint Patterns – How to build effective schemas and endpoints depending on the type of service constructed
    • Chapter 6 – Asynchronous Communication Patterns – A look at options for asynchronous communication and how to exploit this pattern in BizTalk solutions
    • Chapter 7 – Orchestration Patterns – We analyze the role of orchestration in service solutions and see how to build loosely coupled workflow processes
    • Chapter 8 – Versioning Patterns – We see here how to effectively version our SOA solutions and cleanly introduce updates to our BizTalk components
    • Chapter 9 – New SOA Capabilities in BizTalk Server 2009 – A first look at the latest additions to the BizTalk product. This includes the new UDDI v3 registry, WCF SQL Server Adapter, and ESB Guidance 2.0

    So there you go.  Once the book actually hits the shelves, I’ll make another mention of it on the blog.

    Technorati Tags: , ,

  • Presentations from 2009 Microsoft SOA/BPM Conference Available Online

    Hadn’t noticed this before, but found the complete collection of videos and presentations from this year’s SOA & BPM Conference.  I didn’t make it up to Redmond for this, so it’ll be nice to peruse the content which covers topics such as:

    • Customer case studies
    • Designing services for “Dublin”
    • Using BAM for Service and SLA monitoring
    • Supporting WS*, REST and POX simultaneously with WCF
    • SOA patterns from the field
    • Lap around “Oslo”

    Check it out.

    Technorati Tags: , ,

  • Interview Series: Four Questions With … Jesus Rodriguez

    I took a hiatus last month with the interview, but we’re back now.  We are continuing my series of interviews with CSD thought leaders and this month we are having a little chat with Jesus Rodriguez.  Jesus is a Microsoft MVP, blogger, Oracle ACE, chief architect at Tellago, and a prolific speaker.  If you follow Jesus’ blog, then you know that he always seems to be ahead of the curve with technology and can be counted on for thoughtful insight. 

    Let’s see how he handles the wackiness of Seroter’s Four Questions.

    Q: You recently published a remarkably extensive paper on BAM.  Did you learn anything new during the creation of this paper, and what do you think about the future of BAM from Microsoft?

    A:  Writing an extensive paper is always a different experience. I am sure you are familiar with that feeling given that these days you are really busy authoring a book. A particular characteristic of our BAM whitepaper is the diversity of the target audience. For instance, while there are sections that are targeting the typical BizTalk audience, others are more intended to a developer that is really deep with WCF-WF and yet others sections are completely centered on Business Intelligence topics. I think I learned a lot in terms of how to structure content that targets a largely diverse audience without confusing everybody. I am not sure we accomplish that goal but we certainly tried 😉

    I think BAM is one of the most appealing technologies of the BizTalk Server family. In my opinion, in the next releases, we should expect BAM to evolve beyond being a BizTalk-centric technology to become a mainstream infrastructure for tracking and representing near real time business information. Certainly the WCF-WF BAM interceptors in BizTalk R2 were a step on that direction but there are a lot of other things that need to be done. Specifically, BAM should gravitate towards a more integrated model with the different Microsoft’s Business Intelligence technologies such as the upcoming Gemini. Also, having interoperable and consistent APIs is a key requirement to extend the use of BAM to non Microsoft technologies. That’s why the last chapter of our paper proposes a BAM RESTful API that I believe could be one of the channels for enhancing the interoperability of BAM solutions.

    Q: You spoke at SOA World late last year and talked about WS* and REST in the enterprise.    What sorts of enterprise applications/scenarios are strong candidates for REST services as opposed to WS*/SOAP services and why?

    A: Theoretically, everything that can be modeled as a resource-oriented operation is a great candidate for a RESTful model. In that category we can include scenarios like exposing data from databases or line of business systems. Now, practically speaking, I would use a RESTful model over a SOAP/WS-* alternative for almost every SOA scenario in particular those that require high levels of scalability, performance and interoperability. WS-* still has a strong play for implementing capabilities such as security, specifically for trust and federation scenarios, but even there I think we are going to see RESTful alternatives that leverages standards like OpenID, OAuth and SAML in the upcoming months. Other WS-* protocols such as WS-Discovery are still very relevant for smart device interfaces.

    In the upcoming years, we should expect to see a stronger adoption of REST especially after the release the JSR 311 (http://jcp.org/en/jsr/detail?id=311 ) which is going to be fully embraced by some of the top J2EE vendors such as Sun, IBM and Oracle.

    Q: What is an example of a “connected system” technology (e.g. BizTalk/WCF/WF) where a provided GUI or configuration abstraction shields developers from learning a technology concept that might actually prove beneficial?

    A:  There are good examples of configuration abstractions in these three technologies (BizTalk, WCF and WF). Given the diversity of its feature set, WCF hides a lot of things behind its configuration that could be very useful on some situations. For instance, each time we configure a specific binding on a service endpoint we are indicating the WCF runtime the configuration of ten or twelve components such as encoders, filters, formatters or inspectors that are required in order to process a message. Knowing those components and how to customize them allows developers to optimize the behavior of the WCF runtime to specific scenarios.

    Q [stupid question]: Many of us have just traveled to Seattle for the Microsoft MVP conference.  This year they highly encouraged us to grab a roommate instead of residing in separate rooms.  I’ve been told that one way to avoid conference roommates is to announce during registration some undesirable characteristic that makes you an lousy roommate choice.  For instance, I could say that I have a split personality and that my alter ego is a nocturnal, sexually-confused 15th century sea pirate with a shocking disregard for the personal space of others.  Bam, single room.  Give us a (hopefully fictitious) characteristic that could guarantee you a room all to yourself.

    A:  My imaginary friend is a great Opera singer 🙂 We normally practice signing duets after midnight and sometimes we spend all night rehearsing one or two songs. We are really looking have our MVP roommate as our audience and, who knows, maybe we can even try a three-voice song.

    Seriously now, given work reasons I had to cancel my attendance to the MVP summit but I am sure you guys (BizTalk MVP gang) had a great time and drove your respective roommates crazy 😉

    As always, I had fun with this one.  Hopefully Jesus can say the same.

    Technorati Tags: , ,

  • Service Security Guide on MSDN

    The Improving Web Services Security: Scenarios and Implementation Guidance for WCF project on CodePlex now has its results in an online browsable from within the MSDN site.    I linked to this project last year, but it’s great that everything has been made available on MSDN as well.

    Even if you aren’t using WCF, this set of deliverables has some very insightful components.  For example, the Security Fundamentals for Web Services chapter barely even mentions WCF but rather focuses on defining services, overarching security principles, as well as a set of security patterns that address topics such as authentication, data confidentiality and message validation.

    Chapter 2, Threats and Countermeasures for Web Services, is also technology-neutral and identifies a set of security threats, vulnerabilities, and countermeasures.

    Of course it is a WCF guide, so expect to find a wealth of information about WCF security options and trade-offs as well as 20+ “how to” walkthroughs that range from hosting services, to impersonation to using certificate-based authentication.

    Finally, if you’re not a “read tons of pages about security” kind of fella, then at least peruse the WCF Security Checklist (which can provide a good development checkpoint prior to service release), the summary of WCF Security Practices at a Glance (which provides a clean list of categories and related articles) and the very important Q&A section that contains dozens of realistic questions with straightforward answers.

    Great job on this.  Thanks J.D. and team.

    Technorati Tags: ,

  • Should an ODS Have a Query-By-Service Interface?

    My company has an increasingly mature Operational Data Store (ODS) strategy where agreed-upon enterprise entities are added to shared repositories and accessible across the organization.  These repositories are typically populated via batch loads from the source system.  The ODS is used to keep systems which are dependent on the source system from bombarding the source with bulk data load processing requests.  Independent of our ODS landscape, we have BizTalk Server doing fan-outs of real-time data from source systems to subscribers who can handle real-time data events.  My smart architect buddy Ian is proposing a change to our model, and I thought I’d see what you all think.

    So this is a summary of what our landscape looks like today:

    Notice that in this case, our ESB (BizTalk Server) is populated independently of our ODS.  What Ian wants us to do is populate all of our ODSs from our ESB (thus making it just another real-time subscriber), and then throw a “get” interface on the ODS for request/reply operations which would have previously gone against the source system (see below).  Below notice that the second subscriber of data receives real-time feeds from BizTalk, but also can query the ODS for data as well.

    I guess I’ve typically thought of an ODS as being for batch interactions only, and not something that should be queried via request/response operations.  If I need real-time data, I’d always go to the source itself. However, there are lots of benefits to this proposed model:

    • Decouple clients from source system
    • Remove any downstream  impact on source system maintenance schedules
    • Don’t require additional load, interfaces or adapters on source system (although most enterprise systems should be able to handle the incremental load of request/response queries).
    • All the returned entities are already in a flattened, enterprise format as opposed to how they are represented in the source system

    Now, before declaring that we never go to source systems and always hit an ODS (which would be insane), here are some considerations we’ve thought about:

    • This should only be for shared enterprise entities that are distributed around the organization.
    • There is only a “get by ID” operation on the ODS vs. any sort of “update” or “delete” operations.  Clearly having any sort of “change” operations would be nuts and cause a data consistency nightmare.
    • The availability/reliability of the platform hosting the ODS must meet or exceed that of the source system.
    • We must be assured that the source system can publish a real-time event/data message.  No “quasi-real-time” where updates are pushed out every hour.
    • This model should not be used for entities where we need to be 110% sure that the data is current (e.g. financial data, extremely volatile data).

    Now, there still may be a reliance by clients on the source system if the shared entity doesn’t contain every property required by the client.  And in a truly event-driven model, maybe the non-ODS subscribers should only get event notifications and be expected to ping the ODS (which receives the full data message) if they want more data than what exists in the event message.  But other than that, are there other things I’ve missed or considerations I should weigh more heavily one way or the other?  Share with me.

    Technorati Tags: , ,

  • Great New Whitepapers on .NET Services

    One of my early complaints with the .NET Services offerings has been a relative dearth of additional explanations besides the SDK bits.  Clearly just to satisfy me, Microsoft has published a series of papers (authored by those prolific folks at Pluralsight) about the full spectrum of .NET Services.

    Specifically, you’ll find papers introducing .NET Services, .NET Access Control Service, .NET Service Bus, and the .NET Workflow Service.  Upon a quick speed-read through, they all look well thought out and full of useful demonstrations and context.

    Be like me and add these papers to your “to do” list.

    Technorati Tags:

  • BizTalk Server 2009 Misunderstood?

    My buddy Lucas wrote up a thoughtful piece in which he points out some flaws in Zapthink’s opinion on BizTalk Server 2009 and its impact on Microsoft’s SOA strategy.  It’s a good read if you need a refresher on BizTalk’s history and compatibility with SOA principles.  As yes, SOA still matters regardless of the recent chatter about the death of SOA  (Side rant: If you bought a set of products thinking that you were “doing SOA” and your subsequent SOA projects were failures, then you may have missed the point in the first place.  Whether it’s REST, SOA, BPM, etc, it’s the principles that matter, not the specific toolset that will dictate your long term success).

    If you don’t subscribe to Lucas’s blog, you should.  And I need to be better this year about pointing to (and commenting on) some of the great content written by my peers.

    Technorati Tags:

  • Attended Microsoft Cloud Computing Briefing

    Yesterday I was hanging out at the Microsoft office in Los Angeles for an “Architecture Council” meeting on the new Cloud Computing platform.  You can find the slide materials on the Strategic Architecture Council blog.

    Overall, a worthwhile session that you should attend if you get the chance in your city.  My favorite local evangelists were there, including presentations from Woody Pewitt, Kevin Boyle and the delightful David Chou. 

    We discussed what Microsoft considers cloud computing, the trade-offs between “on-premises” and cloud apps (e.g. on-premises apps have more control, customizations vs. cloud apps that are cheaper but sacrifice control and customization).  Also liked the point that we shouldn’t see cloud apps as simply current in-house apps that we shove to the cloud (ala people throwing SOAP interfaces on existing APIs and claiming to be “service oriented”).  We should be identifying a new class of apps that exploit the cloud in a safe, efficient manner.

    Woody presented Live Mesh which I’m aware of, but haven’t spent time actually playing with it.  As with many demos for these new MS tools, the focus was a bit more consumer-oriented.  I’m still trying to get my head around any use cases for a tool like this in enterprise software.

    David presented on .NET Services (formerly BizTalk Services) and once again inspired me to install the latest bits and walk through some scenarios.  Damn him for distracting me from other things.  Finally, we covered Azure and the types of web based apps we can now host in the cloud. 

    David discussed a few use cases for cloud services that got me thinking about how I’d use this in enterprise scenarios.  For instance, it makes a lot of sense to me to expose my organizational’s reference data (product list, sites, etc) as cloud-based services that folks can use in their own apps that collect data points about us.  Why should they maintain their own tables with copies of our reference data if we can provide it in a public internet service? By putting this (data+service) in the cloud, I have one less hole to punch in our external facing infrastructure.

    Clearly lots of this stuff will change prior to the formal release, so for me, these technologies fall into the bucket of “have enough knowledge to be dangerous” and no more.  I need to be able to talk about these technologies and hack up small demonstrations, but most importantly, I want to be able to know when I should consider these technologies in future projects for my company

    Technorati Tags: , ,