Screening: How to select test users for remote user tests

Table of contents

1. Why screening test users is so important?
2. Task of the screener
3. Define the target group
4. Realistically expand the target group
5. Define customer requirements
6. Formulate a screening questionnaire
7. Screening of diligence & motivation
8. Further information for the participants
9. Provide plan B
10. Conclusion: Good screeners are time well spent

1. Why screening test users is so important?

To get meaningful results in your UX / usability tests, you need suitable participants. These should fit the target group of the product you want to test as closely as possible. That sounds obvious. And yet it is not so easy to ensure in practice. There are three main reasons for this:

1. You do not always have access to the right participants.
2. The participants are not selected correctly.
3. Participants do not tell the whole truth. About themselves or their experiences or preferences.

1.1 Problems of portability and credibility

If you only notice that the participants did not pass after the test has run, you will have two problems:

1. The transferability of the results is not guaranteed. Perhaps the users from the actual target group will not have the observed problems at all. Or the users from the target group have completely different problems that did not appear in your test.
2. The credibility of the results is compromised. Even if the results were the same, there is a tendency among stakeholders to consider results with participants who are not entirely suitable as irrelevant. “Our users are smarter.” is sometimes heard here. The better your participants fit into the target group, the easier it is for you to convince stakeholders that the problems you have found also occur in actual use.

1.2 Test users for unmoderated vs. moderated tests

The quality of the test users is particularly important for remote, unmodified tests. Because you do not have a moderator here. In other words, during the test no one notices that the test users may not be suitable. And no one can help either. For example, by providing background information. This allows the participants, for example, to perform tasks that more suitable participants could have performed.

The secondary virtues of your participants also play a role: they must be reliable. In the case of moderated tests, they must go online at the right time. If they are not moderated, they must complete the tasks within the given time. And they need to perform the tasks carefully. This is also particularly important for unmoderated tests since you cannot ask questions if the answers are incomplete or contradictory. However, the participants will not remember correctly afterwards. Sometimes participants make mistakes when filling out the answers. Or when making a screen recording. It is almost impossible to correct this afterwards.

2. Task of the screener

To select whether your potential participants are suitable, you do a so-called screener beforehand. “To screen” means filtering, sifting, or checking. With the screener, you filter out the participants who are most suitable. What you usually also include: Are the candidates motivated to participate? Are they reliable?

The screener is a short online questionnaire. In special cases, you can also speak to the candidates personally. But this is time-consuming and is therefore very rarely done.

It is crucial that you start with the screener as early as possible. Because it often turns out that you do not have access to enough suitable participants. This results in a lot of hectic activity. All personal and professional channels are then used to find further candidates. Or expensive ads are placed to find more people via search engines or on social media. Of course, you can save yourself this hassle if you work with an external recruiter like TestingTime. TestingTime does the screening for you. And provides you with the right, motivated and reliable participants.

3. Define the target group

The first step should not be a lot of effort. However, the client or stakeholder sometimes does not have a clear answer regarding the target group of the product you are testing. If you cannot answer this question, you will never find suitable participants.

Do you get the answer: “Everyone should use our product!” Then it is up to you. Ask yourself:

• Who are the most likely users? Are they more likely to be young people? More tech-savvy? More likely to be women? Employed people? Pensioners?
• Which characteristics, interests or prior experience are important? Which ones play a role in how well they get on with the product? And how interesting do they find it?

If there are personas, all the better. Then you already have a description of the ideal target groups.

In this step, you gather requirements that someone must meet to take the test.

4. Realistically expand the target group

In the next step, you think about which of the criteria just described you can delete. This is because the target group description is almost always too narrow. For example, it says in the persona that this person is single and lives in the countryside. These are rarely criteria that play a role in the test. When it comes to a music streaming service or financial services, marital status and place of residence are almost always irrelevant.

Remember, the closer you define your target group, the more difficult it will be to find the right participants.

5. Define customer requirements

In addition to demographic characteristics such as age, gender, place of residence, income, etc., there are often other requirements. These are mostly prior experiences, habits, or preferences.

For example, this can include:

• Interests, hobbies (mountaineering, art house films, classical music, vegan diet…)
• Habits (visits to the theater, means of transport, sports, media usage…)
• Attitudes (politics, religion, way of life…)
• Use/visiting of certain websites, apps, or services (Spotify, Netflix, bild.de, Freeletics…)

Sometimes you must drop certain requirements because they are too specific. It is therefore good if you prioritize these right away.

An example:

Mandatory requirements

• Between 18 and 45 years old
• Goes mountaineering at least 3 times a year
• Uses at least one tracking or sports app

Optional requirements

• Plans to buy new mountaineering boots in the next 6 months
• Uses the Komoot app

6. Formulate a screening questionnaire

Now we come to the core part, namely formulating the questionnaire for your screening. The following applies here: the best way to reach your goal is often not the direct one. After all, simply asking directly whether the test users meet the criteria is usually not a good idea. First, you want to influence the future participants as little as possible. Therefore, you should not tell them what exactly this is about. On the other hand, if the candidates know which requirements they must meet, then sometimes they don’t tell the truth so that they can take your test.

6.1 Example of a good questionnaire

In this guideline by TestingTime, you will find best practices with concrete examples for creating good questionnaires to identify the ideal people for your study.

6.2 The right balance so as not to reveal too much

You could ask a lot more questions about leisure activities. And you could include other possible answers. But you do not have to know any of that. You are only interested in: Does the candidate go mountain climbing at least 3 times a year? Therefore, you are setting aside so many possible answers that the candidate cannot guess what you are getting at. On the other hand, you make answering as easy as possible for them.

When using the app, it is important not only to ask whether it is used “regularly”. Opinions differ as to what is meant by regular. And asking an open-ended question is too complicated. Users must think about it for too long.

If you have the answers to these screening questions, you will have a pretty good idea of how well the users fit your target group. It is best to use closed questions (i.e. those with predefined answers). This means there is little effort involved in analysing the answers. And the candidates have less trouble answering them.

7. Screening of diligence & motivation

Now there is still one question left which you can ask to test their diligence and motivation. Asking an open-ended question has proven useful for this purpose. Something like this:

Describe in a few words what the ideal app for supporting you in your sporting activities looks like:

That cannot be answered quickly; the test users must think about it. If they only enter 3 words here, they will probably not provide much detailed feedback in your test. And if they respond with something that you do not understand, they will probably have problems later in expressing themselves clearly.

The golden rule for all questionnaires is also very important for screeners: keep it short! The longer your questionnaire, the fewer people will fill it out. And the more disappointed they will be if they are not chosen. This means that in the future they are less likely to answer other questionnaires that you may send them.

KO criteria are a good way of counteracting this. This means that after each question you check whether the candidate is suitable. For example, if they state that they never go mountain climbing, you cannot use them this time. So, there is no need to get them to answer the other questions. Show them a nicely worded closing page right away. And do not let them answer the rest of the questions unnecessarily.

8. Further information for the participants

You should also communicate your expectations of the participants in the screener. For example, they must install an app on their smartphone. They need to make a deposit on a website / app. Or they should make a video recording. Or they should work with a moderator in a video chat on Zoom. There are always people who will have a problem with one requirement or another. Therefore, you should communicate your expectations as early as possible. This saves you and the candidate unnecessary effort.

You also must convey this information:

• When will the candidates find out whether they have been selected and whether they can participate?
• When do they have to participate? (Test date / period during which the questionnaire must be answered)
• What will they get? (Remuneration / Incentive)
• Who can they contact if they have any questions?
• What about data protection?

9. Provide plan B

If someone does not fit 100 percent, do not reject them immediately. You may not get enough participants who are a perfect match. And even if you have enough, do not say no to everyone else. It is always possible that some of them will not show up on the agreed date. Or they will not answer the questions in time. The failure rate (“no show”) is almost always higher with remote tests than with face-to-face tests.

10. Conclusion: Good screeners are time well spent

Creating a good screener and finding the right test users requires a lot of effort. But it is worth paying close attention here. This will ensure that you have less work to do later. And will enable you to achieve meaningful results. And will ultimately ensure that your stakeholders perceive these results as valid and helpful.