Evaluation Criteria for SaaS/Cloud Platform Vendors

My company has been evaluating (and in some cases, selecting) SaaS offerings and one of the projects that I’m currently on has us considering such an option as well.  So, I started considering the technology-specific evaluation criteria (e.g. not hosting provider’s financial viability) that I would use to determine our organizational fit to a particular cloud/SaaS/ASP vendor.  I’m lumping cloud/SaaS/ASP into a bucket of anyone who offers me an off-premises application.  When I finished a first pass of the evaluation, my list looked a whole lot like my criteria for standard on-premises apps, with a few obvious omissions and modifications.

First off, what are the things that I should have an understanding of, but am assuming that I have little control over and that  the service provider will simply “do for me” (take responsibility for)?


Considerations / Questions

  • How do you maintain high uptime for both domestic and international users?
  • What (user and programmatic) interfaces do I have to manage the application?
  • How can on-premises administrators mash up your client-facing management tools with their own?
  • What is the underlying technology of the cloud platform or specific instance details for the ASP provider?
  • What are the storage limits and how do I scale up to more space?
Network configuration and modeling
  • How is the network optimized with regards to connectivity, capacity, load balancing, encryption and quality of service?
  • What is the firewall landscape and how does that impact how we interact with you?
Disaster recovery
  • What is the DR procedure and what is the expected RPO and RTO?
Data retention
  • Are there specific retention policies for data or does it stay in the active transaction repository forever?
  • How are transactions managed and resource contention handled?
Patch management
  • What is the policy for updating the underlying platform and how are release notes shared?
  • How do you handle data protection and compliance with international data privacy laws and regulations?
  • How is data securely captured, stored, and accessed in a restricted fashion?
  • Are there local data centers where country/region specific content can reside?
User Interfaces
  • Are there mobile interfaces available?

So far, I’m not a believer that the cloud is simply a place to stash an application/capability and that I need not worry about interacting with anything in that provider’s sandbox.  I still see a number of integration points between the cloud app and the infrastructure residing on premises.  Until EVERYTHING is in the cloud (and I have to deal with cloud-to-cloud integration), I still need to deal with on-premises applications. This next list addresses the key aspects that will determine if the provider can fit into our organization and its existing on-site investments (in people and systems).


Considerations / Questions

  • How do I federate our existing identity store with yours?
  • What is the process for notifying you of a change in employment status (hire/fire)?
  • Are we able to share entitlements in a central way so that we can own the full provisioning of users?
Backwards compatibility of changes
  • What is the typical impact of a back end change on your public API?
  • Do you allow direct access to application databases and if so, how are your environment updates made backwards compatible?
  • Which “dimensions of change” (i.e. functional changes, platform changes, environment changes) will impact any on-premises processes, mashups, or systems that we have depending on your application?
Information sharing patterns
  • What is your standard information sharing interface?  FTP?  HTTP?
  • How is master data shared in each direction?
  • How is reference data shared in each direction?
  • Do you have both batch (bulk) and real-time (messaging) interfaces?
  • How is initial data load handled?
  • How would you propose handling enterprise data definitions that we use within our organizations?  Adapters with transformation on your side or our side?
  • How is information shared between our organizations securely?  What are your standard techniques?
  • For real-time data sharing, do you guarantee once-only, reliable delivery?
Access to analytics and reporting
  • How do we access your reporting interface?
  • How is ad-hoc reporting achieved?
  • Do we get access to the raw data in order extract a subset and pull it in house for analysis?
User interface customization
  • What are the options for customizing the user interface?
  • Does this require code or configuration?  By whom?
Globalization /  localization
  • How do you handle the wide range of character sets, languages, text orientations and units of measure prevalent in an international organization?
Exploiting on-premises capabilities
  • Can this application make use of any existing on-premises infrastructure capabilities such as email, identity, web conferencing, analytics, telephony, etc?
Exception management
  • What are the options for application/security/process exception notification and procedures?
Metadata ownership
Locked in components
  • What aspects of your solution are proprietary and “locked in” and can only be part of an application in your cloud platform?
Developer toolkit
  • What is the developer experience for our team when interfacing with your cloud platform and services?  Are there SDKs, libraries and code samples?
Enhancement cost
  • What types of changes to the application incur a cost to me (e.g. changing a UI through configuration, building new reports, establishing new API interfaces)?

This is a work in progress.  There are colleagues of mine doing more thorough investigations into our overall cloud strategy, but I figured that I’d take this list out of OneNote and expose it to the light of day.  Feel free to point out glaring mistakes or omissions.

As an aside, the two links I included in the lists above point to the Dev Central blog from F5.  I’ll tell you what, this has really become one of my “must read” blogs for technology concepts and infrastructure thoughts.  Highly recommended regardless of whether or not you use their products.  It’s thoughtfully written and well reasoned.

Technorati Tags: ,

Author: Richard Seroter

Richard Seroter is Director of Developer Relations and Outbound Product Management at Google Cloud. He’s also an instructor at Pluralsight, a frequent public speaker, the author of multiple books on software design and development, and a former InfoQ.com editor plus former 12-time Microsoft MVP for cloud. As Director of Developer Relations and Outbound Product Management, Richard leads an organization of Google Cloud developer advocates, engineers, platform builders, and outbound product managers that help customers find success in their cloud journey. Richard maintains a regularly updated blog on topics of architecture and solution design and can be found on Twitter as @rseroter.

3 thoughts

  1. can this list be applied as it is for evaluating PaaS solutions? openshift/cloudbees and the like.

    1. I would think so. Obviously as you move up from Infrastructure to Platform to Software, the questions change a bit, but overall, I’d hope the categories are similar!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.