As someone who has been working in UX for the past decade, I can’t help but notice that in 2019, our community still has a glaring issue that needs to be resolved: how to demonstrate the return on investment (ROI) of our work.
Of course, we understand and appreciate the impact our work has. But we don’t know how to convince the rest of the company. And this is no trifling matter: we work in political organizations that allocate their limited resources based on perceived importance. So it’s essential that we UXers do the grunt work of crunching the numbers for our stakeholders.
Even in situations where we can’t obtain real data, we can always make a rough estimate. And a rough estimate will always be better than relying on the highest paid person’s opinion (the so-called HIPPO): it’s less subjective and leaves little room for discussion. As Ideo‘s Tom and David Kelley famously said:
“If a picture is worth 1,000 words, a prototype is worth 1,000 meetings.”
Similarly, I posit that a spreadsheet is worth 1,000 baseless arguments with your colleagues.
The ROI of user testing for software teams
There are many methods in the UX toolbox. But in this article, I want to talk about one of the most popular and, to some extent, most misused ones: the user test. The user test gets you the most bang for your buck (check out this instance where a user test revealed that changing a button could increase a site’s annual revenue by $300 Million) and any UXer or product person can use it.
I will add chapters to this article covering the ROI of user testing for hardware teams as well as for marketing and branding purposes. But I want to start with the ROI of user testing for software teams because usually, software teams have the highest levels of UX maturity.
In my experience, there are two main cases in which software teams should conduct user tests: when a product is live and when a new product is being developed. So let’s take a look at these two scenarios in detail.
User testing when a product is live
When a product is up and running, the goals of a user test are to detect and understand problems and to validate solutions. So we need to paint a clear before-and-after picture.
- Start by jotting down hypotheses to build your case.
These will either have to do with saving the company money or increasing the company’s revenue. - Next, figure out your most important KPIs which are linked to your company’s business model:
How does your company make money?
Where does it spend the most?
Once you have a good idea of the areas your hypotheses will impact, gather your data directly from the stakeholders.
Should this approach take too long or not be possible at all, look for benchmarks you can work with instead. - The third and final step is to calculate and compare what it would cost to make an improvement versus what it would cost to do nothing.
Let’s look at a concrete example:
A SaaS company has a subscription-based business model. They offer potential customers a free trial at the end of which only 10% convert to paying customers. The SaaS company hypothesizes that the low conversion rate is due to a poorly designed transition process from free trial to paid subscription: there is no guided onboarding or explanation of the benefits. Instead, potential customers are taken straight to the payment page and required to subscribe in order to regain access to their trial data.
A conversion rate of 10% isn’t terrible. But, of course, there is still room for improvement. Assuming the conversion rate could be 20%, how much is the company currently ‘losing’ by not shooting for this better number?
If 100 people sign up for a free trial, 10 will later become customers, paying a monthly subscription fee of €200 each.
The SaaS company generates €2,000 (10 * €200) a month or €24,000 (12 * €2,000) a year in revenue.
By not attempting to achieve a conversion rate of 20%, the company is currently losing €2,000 (20 * €200 – 10 * €200) a month or €24,000 (12 * €4,000 – 12 * €2,000) a year.
So what would it cost to identify and implement a solution to boost the current conversion rate? Whip out a spreadsheet and make an educated guess:
| Action | Expense |
|---|---|
| Your team will need roughly one week to prepare, conduct, and analyze the first user test. | €2,000 |
| Your team will need to recruit 5 test users. | €500 |
| Your team will need roughly one week to prototype a solution. | €2,000 |
| Your team will need roughly one week to prepare, conduct, and analyze the second user test. | €2,000 |
| Your team will need to recruit 5 test users. | €500 |
| Your team will need roughly three weeks to build and implement the solution. | €6,000 |
| €13,000 |
The return on investment (ROI) is calculated as follows: ROI = (amount gained-amount spent)/amount spent * 100 (to get a figure expressed as a percentage).
So, in the case of the SaaS company, that would be an annual ROI of 85% if they achieve to move forward! Not bad, right?
User testing when a new product is being developed
In this situation, the goal of user testing is to understand the why behind user behaviour. For example, why do your regular customers usually buy from you but sometimes from your competitor? Now is the time to test that backlog full of hypotheses!
Your ROI here is tied to the time it takes your team to develop the new product. So in terms of money: salaries for n people * time spent on the project. (If you know what everyone makes, great. If not, simply use sites like Payscale or Glassdoor for your estimate.)
Let’s assume your product team comprises five people with an average annual salary of €60,000. A two-week sprint would cost you about €2,300 per person so €11,500 total. That means that no matter how you proceed after the first sprint, you will have already burned through €11,500.
Consequently, many companies still opt to develop and launch new products without ever involving the customer in the process. They are attempting to save money, of course. But this strategy is very short-sighted.
Kind of like buying a property you intend to live in without having it inspected first. You could get lucky. Or you could spend your life savings on roofers, electricians, exterminators, and so on.
So, despite the initial added cost, user testing can save you a lot of money in the long run:
| Without user testing | Expense | With user testing | Expense |
|---|---|---|---|
| Sprint (2 weeks) | €11,500 | Sprint (2 weeks) | €11,500 |
| x | x | Recruiting & testing with 5 test users | €2,500 |
| Building & implementing new product (4 weeks) | €23,000 | Building & implementing new product (4 weeks) | €23,000 |
| Reworking final product that does not meet user needs (4 weeks) | €23,000 | x | x |
| €57,500* | €37,000 |
*This is a pretty low estimate. It could take a lot longer than 4 weeks to rework the product. Also, hidden costs like the extra time spent on handling complaints and/or refunds aren’t included.
To calculate the difference in ROI, let’s assume this new product will generate a certain annual revenue.
By making an educated guess about the average order size, for example, we can see the ROI will be lower without user testing, while the ROI will always be greater with user testing:
| Revenue | ROI without user testing | ROI with user testing |
|---|---|---|
| €60,000 | 4% | 62% |
| €80,000 | 39% | 116% |
| €100,000 | 74% | 170% |
The secret benefit of user tests
Until now, I have only been talking about the ROI of user testing in a strictly monetary sense. But there’s a nearly priceless added benefit: improved motivation.
User testing creates a feedback loop, allowing the team to preview how the finished product might (or might not) delight the user once it is launched. Or to put it in startup terms: you get a preview of your traction. This motivates the team and makes everyone feel more connected with the users. No wonder the Silicon Valley Product Group says:
“Behind every great product there is a person with great empathy for the customer.”
All you need is a spreadsheet
I hope this article clarified the ROI of user testing for software teams. You as a UX researcher or designer already have all of the data you need to build an impressive business case and convince your stakeholders. The calculations may seem daunting at first. But, using realistic estimates and a simple spreadsheet, anyone can do it.
By the way, you don’t have to take my word for it. Here’s an amazing TED Talk that explains the concept of simplifying complex situations in a matter of minutes:
Just take a step back, consider the big picture, and then zero in on what really matters. Again, anyone can do it.
Do you have any questions or comments? Please feel free to reach out via email. Do you have an ROI success story you’d like to share with our readers? Please submit your guest post. We look forward to featuring it on our blog.
