Testing Service Oriented Solutions

A few days back, the Software Engineering Institute (SEI) at Carnegie Mellon released a new paper called Testing in Service-Oriented Environments.  This report contained 65 different SOA testing tips and I found it to be quite insightful.  I figured that I’d highlight the salient points here, and solicit feedback for other perspectives on this topic.

The folks at SEI highlighted three main areas of focus for testing:

• Functionality.  This may include not only whether the service itself behaves as expected, but also whether it can easily be located and bound to.
• Non-functional characteristics.  Testing of quality attributes such as availability, performance, interoperability, and security.
• Conformance.  Do the service artifacts (WSDL, HTTP codes, etc) comply with known standards?

I thought that this was a good way to break down the testing plan.  When breaking down all the individual artifacts to test, they highlighted the infrastructure itself (web servers, databases, ESB, registries), the web services (whether they be single atomic services, or composite services), and what they call end-to-end threads (combination of people/processes/systems that use the services to accomplish business tasks).

There’s a good list here of the challenges that we face when testing service oriented applications.  This could range from dealing with “black box” services where source code is unavailable, to working in complex environments where multiple COTS products are mashed together to build the solution.  You can also be faced with incompatible web service stacks, differences in usage of a common semantic model (you put “degrees” in Celsius but others use Fahrenheit), diverse sets of fault handling models, evolution of dependent services or software stacks, and much more.

There’s a good discussion around testing for interoperability which is useful reading for BizTalk guys.  If BizTalk is expected to orchestrate a wide range of services across platforms, you’ll want some sort of agreements in place about the interoperability standards and data models that everyone supports.  You’ll also find some useful material around security testing which includes threat modeling, attack surface assessment, and testing of both the service AND the infrastructure.

There’s lots more here around testing other quality attributes (performance, reliability), testing conformance to standards, and general testing strategies.  The paper concludes with the full list of all 65 tips.

I didn’t add much of my own commentary in this post, but I really just wanted to highlight the underrated aspect of SOA that this paper clearly describes.  Are there other things that you all think of when testing services or service-oriented applications?