There are many important metrics to consider when evaluating cloud computing services. In my book “The Cloud Service Evaluation Handbook: How to Choose the Right Service” (see the interview “The Cloud Market: How to Evaluate and Choose the Best Solution for Your Company“), I listed 56 different metrics, not counting security controls, all of which are optional. That sounds like a lot, but for the very biggest, most strategic decisions you might use them all. For smaller, tactical, project-based decisions you would use a subset. What that subset is depends on the application going into the cloud and the needs of your organization, but I’ll mention a few that would probably be common:
Everyone looks at this, of course, but you have to make sure to include the total cost over time in a way that can be compared. That means including things like transition costs as well as defining a realistic sample configuration for each service and pricing it on each one.
Paying for what you use sounds great until the CFO asks you what next month’s bill will be and you have to say “it depends.” Sometimes that’s okay, but sometimes it’s just not acceptable, and you need a mechanism to control and predict what you’ll be paying.
Is the service going to be up when you need it? When will the provider’s change window be? Is that going to be adequate?
To many people, a cloud service shouldn’t be called cloud unless you can automatically add and remove resources. How long does it take? Is the amount added or removed granular enough?
SLA Targets and SLA Incentives
What happens if the service goes down more than you were expecting, the elasticity doesn’t perform as promised or the response times are too long? Is there any incentive for the provider to avoid those things? Cloud SLAs tend to be less aggressive than they were for non-cloud services. Will they still meet your needs?
As you can see, some of the things you measure for cloud should be the same ones we’ve been using for traditional IT for many years, some have changed and some are completely unique to cloud services. The specific metrics you use for each decision are entirely up to you, but it’s very helpful to have a comprehensive, well-documented master list to choose from, along with detailed instructions on how to use them.
The author: Scott Feuless
Scott is one of the world’s foremost experts on measurement and evaluation of Cloud services. His work with Cloud Computing began in 1999 when he helped transform a small software Value Added Reseller into a public startup Application Service Provider (ASP) as the firm’s Chief Technology Officer. He also joined the Cloud Services Measurement Initiative Consortium (CSMIC), where he was the primary author of the metrics.
Read also his article on how to move to the cloud and avoid costly mistakes.