Building Enterprise Mashups using RSSBus: Part I

I’ve been a fan of RSSBus from /n software for some time. A few weeks ago our Executive Director / Chief Architect / Technology Overlord recently asked me to build a real, live enterprise mashup application to demonstrate for our IT leadership group. Our goal was to show that RSSBus could be used to quickly and efficiently aggregate data in compelling new ways.   In the next few posts, I’m going to walk through our use case, and how I built a solution to solve this.

Before getting into the “what” and “how”, I want to first say that the “how I built a solution” line above isn’t entirely true.  The folks at RSSBus actually built most of the solution for me, and all I did was tweak and customize it a bit.  If you ever get a chance to work with rock stars Ralph James, Amit Sharma or Tom Hearn, jump at it.  I’ll also note that I am working with a pre-release version 2.0 of RSSBus that has new features such as a SOAP connector.

The Problem

As with most large organizations, we have a multiple systems and repositories that contain different aspects of the same data entity.  A “contact” at my company has some attributes stored in our ERP system, some in custom built systems, and some more in COTS applications.  Maintaining a true system of record is difficult in our environment and trying to keep enterprise data repositories in sync is no small task.

Some questions I want answers to …

  • How does a sales rep heading out on a physician visit get a full 360 degree view of the customer that shows EVERYTHING we know about this person?
  • How do I allow people to be proactively notified of any changes to a given sales region or key contact?
  • How do I easily accommodate new data sources (e.g. COTS application, public internet blog or website)?
  • Where can I apply the appropriate security measures to make sure that sensitive contact details are only available for those allowed to see them?  For instance, sales folks may not be allowed to see what clinical trials a physician is participating in, while scientists are not allowed to see marketing activities that a physician has been involved with.

This category of problem can also extend to “complex event processing” scenarios where you want to be able to perform intelligent surveillance on related events that span systems.   However, the problem I’m addressing for us at this moment has to do with data aggregation, not event aggregation.

A Solution

One valid solution to this problem is to use XML feeds (RSS / Atom) from source systems and aggregate them to return a single, complete view of the target data entity.

Why XML feeds instead of a new database repository?  Our reasons include:

  • Subscription model using XML feeds provides users with a great ability to discover, organize and monitor data that sits in multiple places
  • Consumption experience NOT dictated by data generator as end users can choose to eat this data in their feed reader (e.g. Outlook 2007), Excel, SharePoint or custom application
  • Provides a virtual, on-demand data aggregation with minimal changes needed in source systems
  • The pub/sub nature of XML feeds gives users more “sensors” into key data elements in an organization and allows them to receive and act on data in a more timely fashion
  • XML feeds allow an alternate level of service where a query does not have to return real time data or in an immediate fashion

Once I have a feed for each system that stores “contact” details, I want to mash them all up and return a single XML feed that shows an entity whose data comes from all types of data stores.

Now, how do I know which systems this “contact” exists in? At my company, we have a “object registry service” that our applications use to both publish and query enterprise objects.  For instance, we have CRM applications which send “insert” and “update” commands to this service when contact data has changed.  This service is responsible for performing data cleansing on inbound data, and matching inbound objects to existing objects.  The “Richard Seroter” contact inserted by System A should be matched to the “Richard L Seroter” that was already inserted by System B.  What this service stores is only enough information to perform this matching, and, the originating system and primary key.  So, the point is, I can query this service for “Richard Seroter”, and get back all records matching this query, AND, which system (and ID) stores information about this handsome character.

One other wrinkle.  Clearly, many (most?) COTS and custom applications do NOT offer RSS feeds for their underlying data.   So how do I go down this route of XML feeds with systems that don’t natively “talk” RSS? This is where RSSBus comes in.

What is RSSBus?

RSSBus is a lightweight, completely web-based platform for building and publishing XML feeds out of a wide variety of source systems.  It’s a service bus of sorts that takes advantage of the loose contract of RSS to expose and aggregate feeds into interesting business services.

RSSBus uses an impressive array of “connectors” to sources such as Amazon Web Services, PayPal, Microsoft CRM, MySQL databases, Federal Express, Twitter, FTP, LDAP, BizTalk Server and much more.  This means that you can create a feed out of a file directory, or use XML feeds to create new LDAP accounts.   Endless possibilities.

The RSSBus Administration Console can be used to browse connectors, and then use a visual wizard to prototype and generate XML feeds.  You also have full access to the robust RSBScript language which is actually used to generate the raw feeds and optionally, the RSBTemplates which present feed data in an HTML format.  It’s almost a misnomer to call the product RSSBus since data can be emitted not only as RSS, but also ATOM, XLS, CSV, JSON, HTML and SOAP.

Why is RSSBus a good solution for my problem?  Some reasons include:

  • Minimal programming needed to connect to a wide range of mixed platform technologies
  • Strong ability to combine feeds into a single information source
  • Deep scripting and function library for working with and formatting feed data
  • Nice support for feed caching and feed security
  • Can separate data (XML) from presentation logic

RSSBus is NOT a complete XML feed management solution by itself.  That is, RSSBus doesn’t have the full feed discovery and management that an Enterprise Server (NGES) from Newsgator is.  However, note that NGES now plays with RSSBus so that the powerful feeds generated by RSSBus can be managed by NGES.

What’s Next?

In the next set of posts, I’ll look at how I exposed individual RSS feeds from a mix of data sources including Microsoft Excel spreadsheets, databases, web services, and Google queries.  After that, I’ll show you how to mash up the individual feeds into a single entity.  Then, I MAY demonstrate how to apply security and caching aspects to the mashup.

Technorati Tags: ,

Author: Richard Seroter

Richard Seroter is Director of Developer Relations and Outbound Product Management at Google Cloud. He’s also an instructor at Pluralsight, a frequent public speaker, the author of multiple books on software design and development, and a former InfoQ.com editor plus former 12-time Microsoft MVP for cloud. As Director of Developer Relations and Outbound Product Management, Richard leads an organization of Google Cloud developer advocates, engineers, platform builders, and outbound product managers that help customers find success in their cloud journey. Richard maintains a regularly updated blog on topics of architecture and solution design and can be found on Twitter as @rseroter.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.