Integration in the Cloud: Part 3 – Remote Procedure Invocation Pattern

This post continues a series where I revisit the classic Enterprise Integration Patterns with a cloud twist. So far, I’ve introduced the series and looked at the Shared Database pattern. In this post, we’ll look the second pattern: remote procedure invocation.

What Is It?

One uses this remote procedure call (RPC) pattern when they have multiple, independent applications and want to share data or orchestrate cross-application processes. Unlike ETL scenarios where you move data between applications at defined intervals, or the shared database pattern where everyone accesses the same source data, the RPC pattern accesses data/process where it resides. Data typically stays with the source, and the consumer interacts with the other system through defined (service) contracts.

You often see Service Oriented Architecture (SOA) solutions built around the pattern.  That is, exposing reusable, interoperable, abstract interfaces for encapsulated services that interact with one or many systems.  This is a very familiar pattern for developers and good for mashup pages/services or any application that needs to know something (or do something) before it can proceed. You often do not need guaranteed delivery for these services since the caller is notified of any exceptions from the service and can simply retry the invocation.

Challenges

There are a few challenges when leveraging this pattern.

• There is still some coupling involved. While a well-built service exposes an abstract interface that decouples the caller from the service’s underlying implementation, the caller is still bound the service exposed by the system. Changes to that system or unavailability of that system will affect the caller.
• Distinct service and capability offerings by each service. Unlike the shared database pattern where everyone agrees on a data schema and central repository, a RPC model leverages many services that reside all across the organization (or internet). One service may want certificate authentication, another uses Kerberos, and another does some weird token-based security. One service may support WS-Attachment and another may not.  Transactions may or may not be supported between services. In an RPC world, you are at the mercy of each service provider’s capabilities and design.
• RPC is a blocking call. When you call a service that sends a response, you pretty much have to sit around and wait until the response comes back. A caller can design around this a bit using AJAX on a web front end, or using a callback pattern in the middleware tier, but at root, you have a synchronous operation that holds a thread while waiting for a response.
• Queried data may be transient. If an application calls a service, gets some data, and shows it to a user, that data MAY not be persisted in the calling application. It’s cleaner that way, but, this prevents you from using the data in reports or workflows.  So, you simply have to decide early on if your calls to external services should result in persisted data (that must then either by synchronized or checked on future calls) or transient data.
• Package software platforms have mixed support. To be sure, most modern software platforms expose their data via web services. Some will let you query the database directly for information. But, there’s very little consistently. Some platforms expose every tiny function as a service (not very abstract) and some expose giant “DoSomething()” functions that take in a generic “object” (too abstract).

Cloud Considerations

As far as I can tell, you have three scenarios to support when introducing the cloud to this pattern:

• Cloud to cloud. I have one SaaS or custom PaaS application and want to consume data from another SaaS or PaaS application. This should be relatively straightforward, but we’ll talk more in a moment about things to consider.
• On-premises to cloud. There is an on-premises application or messaging engine that wants data from a cloud application. I’d suspect that this is the one that most architects and developers have already played with or built.
• Cloud to on-premises. A cloud application wants to leverage data or processes that sit within an organization’s internal network. For me, this is the killer scenario. The integration strategy for many cloud vendors consists of “give us your data and move/duplicate your processes here.” But until an organization moves entire off-site (if that ever really happens for large enterprises), there is significant investment in the on-premises assets and we want to unlock those and avoid duplication where possible.

So what are the  things to think about when doing RPC in a cloud scenario?

• Security between clouds or to on-premises systems. If integrating two clouds, you need some sort of identity federation, or, you’ll use per-service credentials. That can get tough to manage over time, so it would be nice to leverage cloud providers that can share identity providers. When consuming on premises services from cloud-based applications, you have two clear choices:
  • Use a VPN. This works if you are doing integration with an IaaS-based application where you control the cloud environment a bit (e.g. Amazon Virtual Private Cloud). You can also pull this off a bit with things like the Google Secure Data Connector (for Google Apps for GAE) or Windows Azure Connect.
  • Leverage a reverse proxy and expose data/services to public internet. We can define a intermediary that sits in an internet-facing zone and forwards traffic behind the firewall to the actual services to invoke. Even if this is secured well, some organizations may be wary to expose key business functions or data to the internet.
• There may be additional latency. For some application, especially based on location, there could be a longer delay when doing these blocking remote procedure calls.  But more likely, you’ll have additional latency due to security.  That is, many providers have a two step process where the first service call against the cloud platform is for getting a security token, and the second call is the actual function call (with the token in the payload).  You may be able to cache the token to avoid the double-hop each time, but this is still something to factor in.
• Expect to only use HTTP. Few (if any) SaaS applications expose their underlying database. You may be used to doing quick calls against another system by querying it’s data store, but that’s likely a non-starter when working with cloud applications.

The one option for cloud-to-on-premises that I left out here, and one that I’m convinced is a differentiating piece of Microsoft software, is the Azure AppFabric Service Bus.  Using this technology, I can securely expose on-premises services to the public internet WITHOUT the use of a VPN or reverse proxy. And, these services can be consumed by a wide variety of platforms.  In fact, that’s the basis for the upcoming demonstration.

Solution Demonstration

So what if I have a cloud-based SaaS/PaaS application, say Salesforce.com, and I want to leverage a business service that sits on site.  Specifically, the fictitious Seroter Corporation, a leader in fictitious manufacturing, has an algorithm that they’ve built to calculate the best discount that they can give a vendor. When they moved their CRM platform to Salesforce.com, their sales team still needed access to this calculation. Instead of duplicating the algorithm in their Force.com application, they wanted to access the existing service. Enter the Azure AppFabric Service Bus.

Instead of exposing the business service via VPN or reverse proxy, they used the AppFabric Service Bus and the Force.com application simply invokes the service and shows the results.  Note that this pattern (and example) is very similar to the one that I demonstrated in my new book. The only difference is that I’m going directly at the service here instead of going through a BizTalk Server (as I did in the book).

WCF Service Exposed Via Azure AppFabric Service Bus

I built a simple Windows Console application to host my RESTful web service. Note that I did this with the 1.0 version of the AppFabric Service Bus SDK.  The contract for the “Discount Service” looks like this:

[ServiceContract]
    public interface IDiscountService
    {
        [WebGet(UriTemplate = "/{accountId}/Discount")]
        [OperationContract]
        Discount GetDiscountDetails(string accountId);
    }

    [DataContract(Namespace = "http://CloudRealTime")]
    public class Discount
    {
        [DataMember]
        public string AccountId { get; set; }
        [DataMember]
        public string DateDelivered { get; set; }
        [DataMember]
        public float DiscountPercentage { get; set; }
        [DataMember]
        public bool IsBestRate { get; set; }
    }

My implementation of this contract is shockingly robust.  If the customer’s ID is equal to 200, they get 10% off.  Otherwise, 5%.

public class DiscountService: IDiscountService
    {
        public Discount GetDiscountDetails(string accountId)
        {
            Discount d = new Discount();
            d.DateDelivered = DateTime.Now.ToShortDateString();
            d.AccountId = accountId;

            if (accountId == "200")
            {
                d.DiscountPercentage = .10F;
                d.IsBestRate = true;
            }
            else
            {
                d.DiscountPercentage = .05F;
                d.IsBestRate = false;
            }

            return d;

        }
    }

The secret sauce to any Azure AppFabric Service Bus connection lies in the configuration.  This is where we can tell the service to bind to the Microsoft cloud and provide the address and credentials to do so. My full configuration file looks like this:

<configuration>
<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup><system.serviceModel>
        <behaviors>
            <endpointBehaviors>
                <behavior name="CloudEndpointBehavior">
                    <webHttp />
                    <transportClientEndpointBehavior>
                        <clientCredentials>
                          <sharedSecret issuerName="ISSUER" issuerSecret="SECRET" />
                        </clientCredentials>
                    </transportClientEndpointBehavior>
                    <serviceRegistrySettings discoveryMode="Public" />
                </behavior>
            </endpointBehaviors>
        </behaviors>
        <bindings>
            <webHttpRelayBinding>
              <binding name="CloudBinding">
                <security relayClientAuthenticationType="None" />
              </binding>
            </webHttpRelayBinding>
        </bindings>
        <services>
            <service name="QCon.Demos.CloudRealTime.DiscountSvc.DiscountService">
                <endpoint address="https://richardseroter.servicebus.windows.net/DiscountService"
                    behaviorConfiguration="CloudEndpointBehavior" binding="webHttpRelayBinding"
                    bindingConfiguration="CloudBinding" name="WebHttpRelayEndpoint"
                    contract="IDiscountService" />
            </service>
        </services>
    </system.serviceModel>
</configuration>

I built this demo both with and without client security turned on.  As you see above, my last version of the demonstration turned off client security.

In the example above, if I send a request from my Force.com application to https://richardseroter.servicebus.windows.net/DiscountService, my request is relayed from the Microsoft cloud to my live on-premises service. When I test this out from the browser (which is why I earlier turned off client security), I can see that passing in a customer ID of 200 in the URL results in a discount of 10%.

Calling the AppFabric Service Bus from Salesforce.com

With an internet-accessible service ready to go, all that’s left is to invoke it from my custom Force.com page. My page has a button where the user can invoke the service and review the results.  The results may, or may not, get saved to the customer record.  It’s up to the user. The Force.com page uses a custom controller that has the operation which calls the Azure AppFabric endpoint. Note that I’ve had some freakiness lately with this where I get back certificate errors from Azure.  I don’t know what that’s about and am not sure if it’s an Azure problem or Force.com problem.  But, if I call it a few times, it works.  Hence, I had to add exception handling logic to my code!

public class accountDiscountExtension{

    //account variable
    private final Account myAcct;

    //constructor which sets the reference to the account being viewed
    public accountDiscountExtension(ApexPages.StandardController controller) {
        this.myAcct = (Account)controller.getRecord();
    }

    public void GetDiscountDetails()
    {
        //define HTTP variables
        Http httpProxy = new Http();
        HttpRequest acReq = new HttpRequest();
        HttpRequest sbReq = new HttpRequest();

        // ** Getting Security Token from STS
       String acUrl = 'https://richardseroter-sb.accesscontrol.windows.net/WRAPV0.9/';
       String encodedPW = EncodingUtil.urlEncode(acsKey, 'UTF-8');

       acReq.setEndpoint(acUrl);
       acReq.setMethod('POST');
       acReq.setBody('wrap_name=ISSUER&wrap_password=' + encodedPW + '&wrap_scope=http://richardseroter.servicebus.windows.net/');
       acReq.setHeader('Content-Type','application/x-www-form-urlencoded');

       //** commented out since we turned off client security
       //HttpResponse acRes = httpProxy.send(acReq);
       //String acResult = acRes.getBody();

       // clean up result
       //String suffixRemoved = acResult.split('&')[0];
       //String prefixRemoved = suffixRemoved.split('=')[1];
       //String decodedToken = EncodingUtil.urlDecode(prefixRemoved, 'UTF-8');
       //String finalToken = 'WRAP access_token=\"' + decodedToken + '\"';

       // setup service bus call
       String sbUrl = 'https://richardseroter.servicebus.windows.net/DiscountService/' + myAcct.AccountNumber + '/Discount';
        sbReq.setEndpoint(sbUrl);
       sbReq.setMethod('GET');
       sbReq.setHeader('Content-Type', 'text/xml');

       //** commented out the piece that adds the security token to the header
       //sbReq.setHeader('Authorization', finalToken);

       try
       {
       // invoke Service Bus URL
       HttpResponse sbRes = httpProxy.send(sbReq);
       Dom.Document responseDoc = sbRes.getBodyDocument();
       Dom.XMLNode root = responseDoc.getRootElement();

       //grab response values
       Dom.XMLNode perNode = root.getChildElement('DiscountPercentage', 'http://CloudRealTime');
       Dom.XMLNode lastUpdatedNode = root.getChildElement('DateDelivered', 'http://CloudRealTime');
       Dom.XMLNode isBestPriceNode = root.getChildElement('IsBestRate', 'http://CloudRealTime');

       Decimal perValue;
       String lastUpdatedValue;
       Boolean isBestPriceValue;

       if(perNode == null)
       {
           perValue = 0;
       }
       else
       {
           perValue = Decimal.valueOf(perNode.getText());
       }

       if(lastUpdatedNode == null)
       {
           lastUpdatedValue = '';
       }
       else
       {
           lastUpdatedValue = lastUpdatedNode.getText();
       }

       if(isBestPriceNode == null)
       {
           isBestPriceValue = false;
       }
       else
       {
           isBestPriceValue = Boolean.valueOf(isBestPriceNode.getText());
       }

       //set account object values to service result values
       myAcct.DiscountPercentage__c = perValue;
       myAcct.DiscountLastUpdated__c = lastUpdatedValue;
       myAcct.DiscountBestPrice__c = isBestPriceValue;

       myAcct.Description = 'Successful query.';
       }
       catch(System.CalloutException e)
       {
          myAcct.Description = 'Oops.  Try again';
       }
    }

Got all that? Just a pair of calls.  The first gets the token from the Access Control Service (and this code likely changes when I upgrade this to use ACS v2) and the second invokes the service.  Then there’s just a bit of housekeeping to handle empty values before finally setting the values that will show up on screen.

When I invoke my service (using the “Get Discount” button, the controller is invoked and I make a remote call to my AppFabric Service Bus endpoint. The customer below has an account number equal to 200, and thus the returned discount percentage is 10%.

Summary

Using a remote procedure invocation is great when you need to request data or when you send data somewhere and absolutely have to wait for a response. Cloud applications introduce some wrinkles here as you try to architect secure, high performing queries that span clouds or bridge clouds to on-premises applications. In this example, I showed how one can quickly and easily expose internal services to public cloud applications by using the Windows Azure AppFabric Service Bus.  Regardless of the technology or implementation pattern, we all will be spending a lot of time in the foreseeable future building hybrid architectures so the more familiar we get with the options, the better!

In the final post in this series, I’ll take a look at using asynchronous messaging between (cloud) systems.