Table of contents
What is a usability test?
Why test your product?
What are the different usability tests?
• The traditional moderated test
• Unmoderated remote tests
• UX surveys
• Interviews
• Inhouse tests
Be smart with your time
• Schedule users through surveys
• Combine your moderated user tests and interviews
• Planning is key
Usability testing myths
• Usability testing takes too much time
• “If I were a user…”
• Demographics are not important
• It’s too late to test
Final thoughts
What is a usability test?
Usability tests gather feedback on whether your users can use your product as you designed it. A usability test can be conducted at any time and in multiple ways during your product’s development and when it is launched.
Why test your product?
The honest answer is usability testing saves you and your team time and money. Testing your product early on can bring issues to the surface and give time to rework them with minimal effort. Usability testing can also encourage the team by showing that users like the design and can provide helpful learnings and tips on what works well, what can be improved, and how.
Even when you are short on time (which, let’s be serious, is often the case), usability tests can be done quickly and productively, and save your product from reworks. Below, I have listed the usability methods that I have found the most useful when producing and launching web and mobile games and websites.
What are the different usability tests?
The traditional moderated test
If you already have users you can test with, do it! Users love to give their input. But the trick is to encourage them to be honest and not hold back on criticism. This was the first usability test I took part in when I started my career as a UX Researcher and it taught me a lot about how to approach and engage users, and most importantly, how to treat them with respect. The only downside to this test is that recruiting your users can take a few weeks before you finish testing.
When conducting a moderated test, start by asking some warm-up questions about how your user discovered your product, how often they use it, what other similar products they use, and so on. These warm-up questions can make your users feel at ease and be honest with you. A second researcher can also be useful to have in the room as they can note down responses, actions, and behaviour. If you would like to go one step further, you could also analyse and gain deeper insights into your users’ behaviour and decision-making through eye-tracking technology. Always get permission from your user before recording them.
? Check out Tobii eye-tracking
Unmoderated remote tests
If you don’t yet have users available to test with, then using a recruiting service such as TestingTime is a great way to get user feedback fast. Recruiting services can have your test results ready in as little as a few hours. You will also have more options when choosing your target audience’s demographics, e.g. age, gender, country, computer skills, etc., and you normally have the option to include tailor-made questions.
Another benefit of using a recruiting service is you can see a user’s First Time User Experience (FTUE). This is beneficial because it gives you a fresh look at how users who have never used your product will react.
? More about FTUE
UX surveys
The biggest constraints of usability testing, especially if you are a small team, are cost and time limitations. It is also easy to fall into the trap of spending more time testing and measuring the user experience than improving it. This is where surveys can come in handy. Conducting a survey is an easy, fast and cost-effective way to gather quantitative data, measure user’s feedback over time, and give an indication as to what issues users are experiencing.
Using a standardised survey is a great way to do this. You not only gather feedback but also gauge the usability, experience, and opinions of your product. Standardised surveys can be sent out multiple times over the course of the production phase as well as after the product launch to help monitor ratings and feedback.
Standardised surveys can be sent out multiple times over the course of the production phase as well as after the product launch to help monitor ratings and feedback.
Surveys are, however, not a replacement for moderated and unmoderated remote tests as they cannot provide the same quality of information used to discover the cause of usability issues. They can still provide meaningful data that can help monitor your product’s performance, reflect the user’s experiences, and keep production going.
? More about using surveys to measure UX
Interviews
Although not exactly under the usability test umbrella, user interviews can fill in the gaps that moderated and unmoderated tests and surveys cannot always answer. Interviews are a great way to find out what users want from a product, how they use the product and why, and, most importantly, interviews help your team get to know the users they are designing for. There are some well-known techniques on how to conduct interviews, such as UX laddering or the TEDW technique to help you uncover your user’s core values.
The main problem with conducting interviews is they are time-consuming. Each interview can last anywhere from 30 minutes to more than one hour, and summarising them, let alone transcribing them, can take a while. So if you want to conduct interviews, plan enough time.
? More about conducting interviews
In-house Tests
A quick and dirty method of testing that is free, provides fast feedback, and does not hold up production is testing the product with your colleagues who do not directly work on the product. This is a great way to do a trial run before testing with your users. Your colleagues can familiarise themselves with the product as well as give feedback and already flag any major issues.
It’s important to remember that your colleagues are not your users and your users’ feedback will take priority. Therefore always test with your users after conducting an in-house test.
Be smart with your time
While getting results on surveys and unmoderated remote tests can be done in as little as a day, interviews and moderated tests can take up to a few weeks (sometimes longer). This is because you need time to set up the test, schedule your users, spend on average 30-40 minutes for each interview or test, and write up the results. Here are a few time-saving tips:
Schedule users through surveys
Give users the option to take part in a user test or interview after they have completed a survey. Users love to know that their feedback is taken into account and many are happy to give more input, especially when there’s an incentive. The fastest way to do this would be to set up a microsite where users can go directly from survey to signup, choose their preferred date and time, and voilà! You have your users scheduled.
Combine your moderated user tests and interviews
As mentioned before, asking warm-up questions in a moderated test will help your users feel at ease and give honest feedback. You can also ask users questions, for example: When do they use your product? What motivates them to use it? Why do they keep coming back? And so on. All these questions can help you get to know your users’ needs, habits, motivations, and then you can conduct your moderated test. Job done.
Planning is key
Your users will have questions about your product so you need to know it inside-out. If it’s a game, play it. If it’s a website, explore it. If your product is at the start of its production, talk to the designers, developers, and producers.
Then prepare your usability test by asking yourself the following questions:
- Who is your target audience?
- What exploratory, background, or market research has already been done?
- What previous tests have been done on similar products (if any)?
- What were the learnings?
- When is the product launch/update/release date?
- And finally, which usability test?
Once you are fully immersed in the product and have your plan, it’s time to test.
Usability testing myths
Like any methodology, usability testing is not without its fair share of skeptics and false assumptions. Here are some common ones I’d like to clear up.
Usability testing takes too much time
Testing does take time. But it’s a fraction of the amount of time that usability testing saves. Imagine you went into production without testing and after months of work you have a new, fresh product that nobody can use. This is why testing saves time and money on countless rework rounds that could have been checked earlier in production.
“If I were a user…”
It is beneficial to show the team how a user is using your product. But it is important that the team empathise with the user. More often than not, I hear the words, “If I were a user…” from my colleagues. This means they did not understand why the user did what they did. Always steer the conversation back to what the user actually did rather than what a user would do, and clearly explain to the team why the user carried out that action.
Demographics are not important
Testing the right demographic is vital! Imagine you are testing a new feature of a game. Would you want to test this feature on users who don’t play games? Testing your product with the right demographic will give a clear indication of how your users will behave towards your product. Don’t waste time testing your product with the wrong demographic.
Testing your product with the right demographic will give a clear indication of how your users will behave towards your product.
It’s too late to test
It’s never too late to test because, in the digital world, it’s never too late to change a design. But you still need to budget time for testing. Below is a rough guide on which test to use at each stage of development.
Exploratory, background, and market research
- Method: Interviews with the target audience
- Number of users: 5-30 users
- When to test: Pre-production
- Duration: 2-5 weeks
- Rework: None
Wireframes or skeleton designs
- Method: Unmoderated remote tests (FTUE) or, if time permits, moderated tests (own users)
- Number of users: 5-12 per group
- When to test: During production, as early as possible
- Duration: 1-2 days for unmoderated remote tests (15-30 mins per user), 2-5 weeks for moderated tests
- Rework: Any changes will require a minimal amount of time and effort
Prototypes (low- & high-quality)
- Method: Unmoderated remote tests (FTUE) and, if possible, moderated tests (own users)
- Number of users: 5-12 per group
- When to test: During production, after testing wireframes
- Duration: 1-2 days (15-30 mins per user) for unmoderated remote tests, 2-5 weeks for moderated tests
- Rework: Minimal, if low-quality. If high-quality or final graphics are already implemented, reworking could take longer.
Minimal Viable Product (MVP)
- Method: UX surveys and, if possible, moderated tests (own users)
- Number of users: 300-400 users for UX surveys, 5-12 users for moderated tests
- When to test: Near the end of production, before launch
- Duration: 1-3 days for UX surveys, 2-5 weeks for moderated tests
- Rework: Minimal rework as major issues should have already been corrected
Product launch
- Method: UX surveys and, if time is available, interviews
- Number of users: 300-400 users for UX surveys, 5-30 users for interviews
- When to test: After the launch
- Duration: 2-4 days for UX surveys, 2-5 weeks for interviews
- Rework: Small tweaks
Final thoughts
Conveying results of usability tests, especially if there are major issues, is not always easy and it can take time for your colleagues to see the value in testing. But keep at it, keep looking for opportunities to test, and find ways to compromise for time. Remember: your users are your heroes. They are the ones who will help make your product amazing!
