It may seem a bit strange, but recently I’ve been asked about Return on Investment (ROI) calculators a fair amount. Generally the discussion is if a given vendor should (or should not) have an ROI calculator on their website. Some reasons for justifying an ROI calculator include making it easier for potential clients to understand the value they will receive from a solution and the basic cost involved. Of course, as with many things, the above reasons, while all perfectly valid, suffer from basic faults.
While ROI calculators can be a good starting point, you will never get a highly accurate estimate of ROI by punching numbers into a web form. Environments are way too complicated to where a few variables can be used to estimate accurate returns. To get a truly accurate calculation, a system would need to account for thousands of variables in a given environment. And missing just one could render the entire calculation useless.
Understanding the basic cost of something is always helpful. Back when I was involved in procurement, it always made me crazy when vendors refused to provide ballpark price estimates. I worked for a tiny company then and needed to know “is this going to cost $10k or $100k?” We could afford $10k. But it would be a waste of time sitting through a full sales presentation if the ballpark cost was $100k.
From most vendor perspectives, any kind of pricing from an ROI calculator is really just a wild guesstimate. I think the exercise can be useful, because you get at least some sort of basis for cost. But there are other things in play, like sales discounts, accurate counts of license needs, etc. Therefore, it’s important to remember the numbers generated are really just guesstimates.
The Limitations of ROI Calculators
ROI calculators are, by their nature, limited to only a few data points (likely ten or fewer) and have to generalize across organizations and industries. This means that for any given organization, an ROI calculator might be accurate. However, ROI calculators are just as likely to be off by some significant margin. I haven’t done any math on this, but I suspect you’ll all agree.
Before you put too much faith in the estimate from the ROI calculator, ask yourself a few questions:
Is the information they are basing the calculation on valid for you and your organization?
Does the calculator take into account things like organization transitions, digital transformation, internal company politics, etc? Many organizations are unlikely to achieve the full benefit listed in an ROI calculator due to internal politics. If a new tool is being brought in that will save $1,000,000 dollars per year with a full, company-wide deployment according to the ROI calculator – but upper management refuses to fully deploy – full savings will not be realized. It is easy to say something like “that isn’t the calculator’s fault” and that is true. But it also undercuts the benefits of the ROI calculator. I may have told my boss the company is going to save $1,000,000 when in reality, we only saved $100,000.
Making an adjustment for specific verticals is helpful, but even within specific verticals organizations differ. The amount of variation, well, varies, but there is always some. This, unfortunately, leads to less accuracy from the ROI calculators.
Like real estate, if you are in the healthcare industry, for example, location matters a lot. Whether you are a major teaching hospital in a metropolitan area or a small, rural hospital is going to make a big difference on the data points that are important for calculating your ROI. A big organization is likely to have a large staff, so a complicated tool may make sense for them. Larger organizations have the people to learn, support and deal with the tool on a daily basis. Smaller organizations (not tiny, just smaller) may not have the staff to support complex tools. Thus, ease of use is the most important factor.Therefore, just making adjustments for specific industries or areas won’t work, without explanation. If an ROI calculator doesn’t explain what adjustments are actually being made – and let’s face it, I’ve never seen such detail in an ROI calculator – those entering the data don’t actually know what factors are taken into account differently.
What assumptions are they making about the industry, the organization, the product, etc.?
Every calculator has to make assumptions. As mentioned above there are thousands of data points impacting your organization, but the calculator can only take into account a few of them. Even if these assumptions are highly accurate for your organization (and that’s highly unlikely) different assumptions are also being made about your industry in general. If these assumptions are wrong, it can throw off the entire calculation.
Is the calculator taking into account things like staffing requirements, physical hardware, connectivity costs, on-going maintenance, and other “non-standard” data points?
Many calculators focus on the specifics of the solution and solely from the perspective of the solution. Unfortunately, this is very much like high school physics, where they tell you to do things like ignore air resistance. As an aside, check out a great example of this concerning hail stones! Sure, you can ignore air resistance (or friction, or whatever it is that is inconvenient), and things work on paper. But, when you try to test in real-world environments, you can’t simply ignore some factors.
Calculators that do not account for having to hire new employees, having to acquire and deploy physical boxes, pay for increased bandwidth (if you are using a cloud or SaaS tool for example) and other “non-standard” costs, mean you are potentially missing out on a significant amount of information that is likely to drastically impact the overall ROI.
Is the ROI calculator actually showing you useful information?
ROI calculator’s love to show how much money you will either save or make after deploying a vendor’s solution. While this information may be useful, it is important to track the information the calculator is providing with the metrics you care about, are responsible for, or need to deal with. If the metrics you need are not available from the calculator, how useful is the ROI calculation in reality?
In Conclusion
ROI calculators can be immeasurably helpful to both the vendor and the customer, allowing both to get a better understanding of deployments, costs, savings, etc. It is critical, however, to take the information as it is, an estimate. Remember, the numbers given could be right, wrong or more than likely, somewhere in between.
What’s more, ROI depends greatly on the value you get from our ITIM tool, and that points to how you use the data your tool captures. Check out our new eBook – Monitoring and Observability: Making Metrics Matter – for an in depth view on how IT can drive business results leveraging Netreo’s ITIM solution.