I’ve been a fan of RSSBus from /n software for some time. A few weeks ago our Executive Director / Chief Architect / Technology Overlord recently asked me to build a real, live enterprise mashup application to demonstrate for our IT leadership group. Our goal was to show that RSSBus could be used to quickly and efficiently aggregate data in compelling new ways. In the next few posts, I’m going to walk through our use case, and how I built a solution to solve this.
Before getting into the “what” and “how”, I want to first say that the “how I built a solution” line above isn’t entirely true. The folks at RSSBus actually built most of the solution for me, and all I did was tweak and customize it a bit. If you ever get a chance to work with rock stars Ralph James, Amit Sharma or Tom Hearn, jump at it. I’ll also note that I am working with a pre-release version 2.0 of RSSBus that has new features such as a SOAP connector.
The Problem
As with most large organizations, we have a multiple systems and repositories that contain different aspects of the same data entity. A “contact” at my company has some attributes stored in our ERP system, some in custom built systems, and some more in COTS applications. Maintaining a true system of record is difficult in our environment and trying to keep enterprise data repositories in sync is no small task.
Some questions I want answers to …
- How does a sales rep heading out on a physician visit get a full 360 degree view of the customer that shows EVERYTHING we know about this person?
- How do I allow people to be proactively notified of any changes to a given sales region or key contact?
- How do I easily accommodate new data sources (e.g. COTS application, public internet blog or website)?
- Where can I apply the appropriate security measures to make sure that sensitive contact details are only available for those allowed to see them? For instance, sales folks may not be allowed to see what clinical trials a physician is participating in, while scientists are not allowed to see marketing activities that a physician has been involved with.
This category of problem can also extend to “complex event processing” scenarios where you want to be able to perform intelligent surveillance on related events that span systems. However, the problem I’m addressing for us at this moment has to do with data aggregation, not event aggregation.
A Solution
One valid solution to this problem is to use XML feeds (RSS / Atom) from source systems and aggregate them to return a single, complete view of the target data entity.
Why XML feeds instead of a new database repository? Our reasons include:
- Subscription model using XML feeds provides users with a great ability to discover, organize and monitor data that sits in multiple places
- Consumption experience NOT dictated by data generator as end users can choose to eat this data in their feed reader (e.g. Outlook 2007), Excel, SharePoint or custom application
- Provides a virtual, on-demand data aggregation with minimal changes needed in source systems
- The pub/sub nature of XML feeds gives users more “sensors” into key data elements in an organization and allows them to receive and act on data in a more timely fashion
- XML feeds allow an alternate level of service where a query does not have to return real time data or in an immediate fashion
Once I have a feed for each system that stores “contact” details, I want to mash them all up and return a single XML feed that shows an entity whose data comes from all types of data stores.
Now, how do I know which systems this “contact” exists in? At my company, we have a “object registry service” that our applications use to both publish and query enterprise objects. For instance, we have CRM applications which send “insert” and “update” commands to this service when contact data has changed. This service is responsible for performing data cleansing on inbound data, and matching inbound objects to existing objects. The “Richard Seroter” contact inserted by System A should be matched to the “Richard L Seroter” that was already inserted by System B. What this service stores is only enough information to perform this matching, and, the originating system and primary key. So, the point is, I can query this service for “Richard Seroter”, and get back all records matching this query, AND, which system (and ID) stores information about this handsome character.
One other wrinkle. Clearly, many (most?) COTS and custom applications do NOT offer RSS feeds for their underlying data. So how do I go down this route of XML feeds with systems that don’t natively “talk” RSS? This is where RSSBus comes in.
What is RSSBus?
RSSBus is a lightweight, completely web-based platform for building and publishing XML feeds out of a wide variety of source systems. It’s a service bus of sorts that takes advantage of the loose contract of RSS to expose and aggregate feeds into interesting business services.
RSSBus uses an impressive array of “connectors” to sources such as Amazon Web Services, PayPal, Microsoft CRM, MySQL databases, Federal Express, Twitter, FTP, LDAP, BizTalk Server and much more. This means that you can create a feed out of a file directory, or use XML feeds to create new LDAP accounts. Endless possibilities.
The RSSBus Administration Console can be used to browse connectors, and then use a visual wizard to prototype and generate XML feeds. You also have full access to the robust RSBScript language which is actually used to generate the raw feeds and optionally, the RSBTemplates which present feed data in an HTML format. It’s almost a misnomer to call the product RSSBus since data can be emitted not only as RSS, but also ATOM, XLS, CSV, JSON, HTML and SOAP.
Why is RSSBus a good solution for my problem? Some reasons include:
- Minimal programming needed to connect to a wide range of mixed platform technologies
- Strong ability to combine feeds into a single information source
- Deep scripting and function library for working with and formatting feed data
- Nice support for feed caching and feed security
- Can separate data (XML) from presentation logic
RSSBus is NOT a complete XML feed management solution by itself. That is, RSSBus doesn’t have the full feed discovery and management that an Enterprise Server (NGES) from Newsgator is. However, note that NGES now plays with RSSBus so that the powerful feeds generated by RSSBus can be managed by NGES.
What’s Next?
In the next set of posts, I’ll look at how I exposed individual RSS feeds from a mix of data sources including Microsoft Excel spreadsheets, databases, web services, and Google queries. After that, I’ll show you how to mash up the individual feeds into a single entity. Then, I MAY demonstrate how to apply security and caching aspects to the mashup.
Richard Seroter's Architecture Musings 
4 thoughts