9 golden rules for solid user testing

The aim of this article is to make you aware of the basics of good user testing and to show you how important it is. To this day, qualitative user research is repeatedly accused of not being able to measure success or failure sustainably – unlike quantitative research methods. This makes it all the more important to proceed in a methodologically correct manner and to observe set rules and best practices.

A moderated user test involves a preparation, implementation and analysis phase. On average, a complete user test takes three to four days. This period is typically divided as follows:

• 1-2 days preparation
• 1 day of execution
• 1 day analysis and reporting

At this point, I would like to discuss the three most important rules for each of these phases.

Overview of the nine rules of user testing:

Preparation:

1. Add a hypothesis and a goal to the test questions in the test script.
2. Each usability test requires five to seven test persons.
3. Perform test run.

Execution:

1. Encourage your test users to think aloud.
2. Don’t ask suggestive questions.
3. The only people speaking during the test are the moderator and the test user.

Analysis:

1. Don’t wait too long to do the evaluation.
2. Don’t jump to conclusions based on individual statements.
3. The report should also include positive feedback.

Preparation

Good preparation is half the battle. However, people often underestimate what preparation really entails. The time required for this phase is one to two days. The most important part is to understand the objective of the test. Usually the starting point is a prototype or an existing website/app. For example, your job might be to clarify whether a sub-function needs to be checked for user-friendliness or whether the entire website needs to be assessed. It is also important to define where the test will be performed (e.g. office, lab, outdoors) and who the target audience is. This will help you to formulate the questions accurately and to be more specific when it comes to recruitment. Part of the preparation is writing a test script and performing a test run with the prototype. During the preparation phase, I recommend observing the following three rules:

1. Add a hypothesis and a goal to the test questions in the test script.

This way, you can anticipate what might happen and at the same time be better prepared for the test (e.g. the test user will not discover the search field in the footer as intended). The test script is the moderator’s guiding thread and is not shared with the test user.

2. Each test requires five to seven test users.

There is probably no better explanation on this subject than the study by Jakob Nielsen. In addition, it should be possible to complete a user test in one day. More than seven people are difficult to manage both mentally and in terms of time. Depending on the target group segmentation, it can be worthwhile to test several times with five to seven people.

Source: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

3. Perform test run.

A malfunctioning prototype or an illogical test script can render a test useless. These 60 minutes are time well spent and will save you a lot of time later. It is important that the developer of the prototype is present to make any improvements if necessary. In order to ensure that all test users have the same starting conditions, it is also important to make sure that the test script and the prototype will not need to be adapted during the execution. In order to help you write test scripts even faster in the future and to make sure you don’t forget anything, we have prepared a guide with worksheet templates for you.

Implementation

The moment of truth. The execution is the most intensive part of the user test. Depending on the design of the test, the execution will take place in the office, remotely or in a usability lab. If the test is taking place at your office, I recommend that you read the five golden rules for advanced user testing to ensure that the test users have a positive experience. Always be aware that all test users are different and learn to respond to and understand their different personalities. I therefore recommend that you never take the test alone. Have an observer/notetaker so that you can better concentrate on moderating. To ensure the test is successful, it is essential that you follow the following three rules during this phase:

1. Encourage your test users to think aloud.

This may feel a little unnatural to the user, but if you can’t understand exactly what’s going on in the test user’s mind, you won’t be able to learn anything. If the user starts thinking quietly, you need to ask them to think out loud again. There are some usability experts who find loud thinking unnecessary because it forces the test user to behave unnaturally. Personally, I find this method extremely important.

Thinking aloud: Test user doing a usability test

2. Don&’t ask suggestive questions.

This is a classic among market researchers. Asking the right questions is a real art. Often, it can be helpful to involve an unbiased moderator who is only marginally or not at all involved in the project. This prevents the test user from being steered in the direction you might be hoping for. The test run also offers a good opportunity to check and practise whether suggestive questions are being asked or reactions are being provoked.

3. The only people speaking during the test are the moderator and the test user.

Observers need to wait until the end of the test to ask questions. Interrupting questions interfere with the process. For this reason, it is important to point out clearly before each test that questions may only be asked at the end. This is particularly important if there are important stakeholders in the room who would like to comment on the whole thing. In such cases, it is advisable to have observers sit in a separate room. In order to streamline the subsequent analysis, it is very important that the observers write legible notes. To ensure that you can make the most of your notes at the end of the test, we have written sample note sheets for you.

Analysis

Immediately after the execution comes the analysis. The goal is to obtain usable and prioritised action items. The results form the basis for all further steps in the design and development process of your product or service. As a general rule: Anything that contributes to improving a product or service should be addressed. Of course, there are always things that can be solved with more or less effort. When it comes to analysis, I recommend focusing on the following three rules:

1. Don’t wait too long to do the evaluation.

The best practice is to carry out a debriefing with all participants on the same day and record the most important findings in a table. If you wait too long with the analysis, you may have to look through all the videos again. This will unnecessarily prolong the process. Videos serve to provide answers to detailed questions, but not to show basic problems after the fact.

2. Don’t jump to conclusions based on individual statements.

This rule also applies to the execution. Often, people discuss a solution immediately after a single test, even though they do not yet know how others will react to it. I, therefore, recommend recording the findings first and waiting until the end of the day to see how often the problem has occurred.

3. The report should also include positive feedback.

As UX designers, we focus on identifying problems. After all, this is the very essence of our work. But the report’s audience will be more susceptible to criticism if there is a healthy portion of praise. The praise should be honest, and every tested object will have at least one positive aspect. We recommend using TestingTime’s user test analysis spreadsheet in the ux testing kit. It will show you quickly and easily which points are important and which are not. Of course, there are many more additional rules, but they are beyond the scope of this article. If you are interested in brushing up your skills and knowledge for conducting user tests, I recommend our guide to moderated user tests. It elaborates on all the points mentioned above and covers many more aspects of successful user testing.

Conclusion

Adhering to the nine rules discussed above dramatically increases your chances of getting more relevant results from your user test. Of course, this does not guarantee that your product or service will automatically be more successful on the market. It is important that the rules are applied on a regular basis and for each user test. If these rules are applied in your day-to-day activities, the credibility of your methodological skills will increase and so will the credibility of your results. Now we want to hear from you: Have you already internalised the rules or do you use completely different guidelines when testing? Share your thoughts with us in the comments. Good luck with your next user test! To order users for your user tests quickly and easily, click here.