Recruitment is one of the key issues in UX research. Recruitment decisions can impact how projects unfold and it is generally understood that testing with users in the target audience yields the best results. However, in some cases, testing outside the target demographic makes more sense.
It is a logical and correct assumption that user tests with target-audience users are best. If you are not clear on who your target audience is or what, when, and how they are going to use your product or service, you can end up with skewed research results that are not valid for your specific target audience. Consequently, in the discovery phase, it is crucial that you get to know your target audience and their specific needs. In this phase, you must necessarily interview users who match your specific target demographic!
But the always-test-your-target-audience mantra does not apply to each and every situation. Think of other phases such as development. Here, you should question whether testing with your target audience is a necessary or even a reasonable option.
Bending the rules
But why not always test with your target audience, you ask? Testing with the target audience is, indeed, best. If your target audience is easy to come by – say, if you are developing a pizza delivery app aimed at a good portion of the population – good for you. You can carry out all of your tests using your target audience.
However, some target demographics can be quite difficult to come by. Think about people who are rare, earn a lot, or are incredibly busy and important. Heart or brain surgeons, for example. In these cases, recruiting test users in your exact target audience may take a long time or cost a lot of money. Testing solely with these people would mean testing later, testing less, and not iterating very often. And, as a consequence, your product or service would have inferior usability compared to if you had tested more frequently with another group.
So it is worth considering other options. Namely, splitting your research up into two separate steps:
- Carrying out usability tests with other people who are easier to recruit.
- Testing with your target audience only when this is truly necessary.
The reason I recommend this strategy is: specialist knowledge does not help users to navigate a website. If a standard user cannot find a button to proceed to the next step on a website, neither will a heart or brain surgeon. So concentrate on getting the kinks out first. Then perform a trial run with real future users.
Age and culture always matter
We already looked at why testing with your specific target audience is important in the discovery phase. But this changes as we enter the development phase. At this point, we are interested in a product or service’s usability. An app or a website, for instance, should be easy and intuitive for just about anyone to use.
Never test products for digital natives with digital immigrants or vice versa.
That said, you do need to zero in on two criteria when you’re recruiting easier-to-get-ahold-of test users for your study:
1 Age
Always use participants who fall into the same age groups as the people in your target audience. Digital natives and digital immigrants have highly different characteristics when it comes to how they approach technology. Digital natives grew up with technology. They use countless apps every day and spend a lot of their lives online. Thus, they have a lot of experience with all kinds of standard interaction elements.
But digital natives and digital immigrants also differ very much with regard to figuring out how something works. Digital natives go ahead and try things out in order to see what happens. Digital immigrants, on the other hand, are often hesitant. They prefer to think about how things might work before deciding on a particular course of action. While this is not meaning to say that one approach is better than the other, it certainly demonstrates how a digital immigrant might need more cues to identify what the next correct step is. Therefore, if you test a product that is meant for digital immigrants with digital natives, you could end up with skewed results.
2 Culture
The most obvious reason to test a product or service in the country it is being developed for is that the locals will speak a particular language. Translations often result in misunderstandings and so an app, for example, that worked well in one country should not simply be translated and then launched in another country without proper testing.
Furthermore, different cultures have different preferences, for example, with regards to the types of colours and images they consider appropriate in a certain context. But concepts, too, like what is considered modern- or oldfashioned-looking vary strongly.
Much more surprisingly, not even usability is constant across cultures. This is because different regions have different device preferences (such as preferred screen size) and web standards, resulting in different usability issues from country to country.
By the way, I am not talking about large distances across continents here. There are immense discrepancies between geographically close countries like Germany and the UK. So even if you are solely testing usability, you must test in the region in which you plan to launch your product or service.
Always test preferences with your target audience
In the development phase, you may also be interested in things beyond usability. For example, whether the imagery or the content of your website appeals to your target audience. In this case, if you test outside the target demographic, you are basically asking the wrong people to guess at what the right people will like. Obviously, this makes no sense. You need to test preferences with your target audience. If, on the other hand, usability is the only thing you need to test, there are very few cases in which you need to do this with your target audience:
- In some cases, you may find that your target audience is used to certain standards that other people would not be used to. This could be because your target audience uses a particular, industry-specific website or software.
- Products and services for the disabled are also obvious examples. You must always test their usability with people who have exactly those disabilities the product or service caters to.
How to decide
So how do you decide that testing outside the target demographic is okay? When you can answer ‘yes’ to both of these questions:
- Are you interested solely in usability (i.e. no preferences, appropriateness of content, etc.)?
- Does your target audience have the same (computer) habits and skills as other users?
Remember that you will still need to recruit test users who are similar to your target audience, i.e. they should belong to the same age group and culture as your target audience. Another rule of thumb to bear in mind has to do with (computer) skills: always test users with lower rather than higher skills. Why? Because users with higher skills will be able to use products and services that users with lower skills are able to use. But you cannot assume that the opposite will hold true.
Now, let’s put all of that theory into practice with an example.
A Wikipedia-type website
Let’s assume you are working on a medical, Wikipedia-type website aimed at doctors and medical researchers. It will provide them with a quick and informal way to record and exchange information with their colleagues.
You know from informal conversations at medical conferences that there is a strong need for this to exist. Additionally, you have got the financial support to develop the website. But you still do not know enough. When and how often would doctors and medical researchers use such a tool? Which devices would they access it with? Where would they use it? What features would it need to have? And so on. This is a case where you should test with your target audience. Recruit people who match your criteria exactly and find out what they need with a focus group or individual interviews.
Once you have started to develop the website and have built the first prototype, it is time to collect some feedback. How intuitively can the website be used? Do the structure and the medical categories make sense? Some medical knowledge is required to answer the latter question. Therefore, again, your first thought might be that you must necessarily research this by testing people in your target audience.
Consider other users with similar traits
However, you could also consider testing your medical categories with other medical professionals who are easier to get ahold of first. These people could be advanced medical students, nurses, or healthcare practitioners. Anyone with enough medical training to evaluate the logic behind basic medical categories. After the first round of tests with this group of people, you will only need very few doctors and medical researchers, i.e. people in your target audience, to be able to validate the structure and the categories of your website.
Validate with people in your target audience.
With regard to the question of how usable your prototype is, you could start testing this with users who are quite easy to attain. Even more so than people with a medical background. Simply focus on recruiting test users who belong to the same age group and who come from the same country as your target audience. Additionally, if you know that, for example, your target audience uses certain devices more often than others, include that criterion in your recruitment process.
Test early, test often
Once you have built the next iteration of your prototype, you can test it again with a similarly attainable group of people. Afterwards, as a first milestone, you can test with a few people from your target audience and then go back to the more attainable group for further development. Continue to do this over and over again and you will end up needing way fewer hard-to-recruit test users overall. You will be able to test earlier on and more often. And most importantly, you will make sure that you stay on the right track. Saving your employer a lot of money in the long run!
In conclusion, testing with the target demographic is great. But some test users are so difficult to get ahold of that you need to consider other options. You do not want to have to settle for fewer tests that can only start late in the development process. If your budget or a lengthy recruitment process is holding you back, try splitting your topics up. There is no need to test absolutely everything with your target audience.
An agile development style demands that you test early and test often. Waiting until the week before the release to perform one large, expensive test is not a valid option. So do not shy away from making this somewhat controversial recruitment decision where appropriate. After all, a medical degree will not help a user to find their way back to the home page.
