How to conduct remote user tests successfully

Table of contents

1. Online is not just online
1.1 Moderated online usability tests
1.2 Unmoderated online usability tests
1.3 Advantages of online tests
1.4 Disadvantages of online tests

2. How do I deal with the limitations of moderated tests?

3. How do I deal with the fact that I cannot react without moderation?
3.1 Pay attention to good test users
3.2 Think of good tasks
3.3 Write a top briefing
3.4 Always, always do a trial test
3.5 Check first answers from test users
3.6 Recruit more participants
3.7 Limitations for tests with mobile devices
3.8 Screen recording from iPhone, Android & Tablet
3.9 Alternatives to tests with mobile devices
3.10 Purposes/questions for online tests & efforts

4. Summary

1. Online is not just online

There are several types of online tests. A differentiation is fundamental. Why? Because it determines not only how you work, but also what you can test in the first place. This differentiation is between moderated and unmoderated.

1.1 Moderated online usability tests

Moderated online usability tests are in the end classic usability tests in which you simply do not sit in the same room with the test user. If you have already done a few tests in the usability lab, you will hardly have any problems with moderated online tests. Since you and your test user work simultaneously, these tests are also called synchronous remote usability tests (sRUT).

1.2 Unmoderated online usability tests

The unmoderated test, on the other hand, is called asynchronous remote usability test (aRUT). The test is carried out in such a way that the test users receive their test tasks in writing from you. They then work through the test independently. They are independent of you as the moderator. Therefore, such tests are called “asynchronous”.

1.3 Advantages of online tests

The main advantages of online tests are:

• Your test users do not have to make the journey to the test location.
• They can test in their familiar environment on their own devices.
• You can schedule the tests much more flexibly.
• You do not need a usability lab or a quiet office for testing.
• Observers at live tests can easily participate. They see and hear exactly the same as you do as the moderator.
• A single test is cheaper (less time required, less travel costs, no rent, lower incentives/payment of test users).
• A single test is faster.
• You can test with more people or recruit test users who are otherwise hard to get.

And finally, with unmoderated online tests, you have the advantage that all test users actually have exactly the same conditions. The distortion of the results due to different moderation is therefore eliminated.

The general procedure of an unmoderated online usability test from the user’s perspective.

1.4 Disadvantages of online tests

Of course online tests also have disadvantages. Some UX colleagues find them so serious that they do not want to work with them at all. This is understandable, since user tests are mainly about getting in touch with real users. And this is limited in all online tests.

Disadvantages of moderated online tests

So let’s look at the disadvantages of moderated online tests compared to their on-site counterparts:

• Getting to know and chatting with users in a casual way is not so easy via video.
• In the usability lab you will notice very quickly if a test user becomes impatient or does not feel well. With a video broadcast this is not so easy to detect.
• If you sit next to a test user, you can control him or she with your own body language. By bending over and nodding your head, for example, you signal interest and encourage the test user.
• The test users may be distracted by colleagues, roommates, children or pets. Sometimes you don’t even notice that. This can also be an advantage – you may have more realistic terms of use.
• Maybe you are testing things that should not (yet) be made public. In this case, the only protection you have is to have the test users sign confidentiality agreements. You can’t technically ensure that nobody takes a screenshot or a photo with the smartphone.

User test in the usability lab.

Disadvantages of unmoderated online tests

It is even more difficult with unmoderated tests. Because with these you have nothing to do with the users. This causes the following additional difficulties:

• The test users need more technical expertise.
• You have no possibility to react to unforeseen errors and problems. For example, if there is a technical problem, the test is over.
• You cannot control the course of the test if it takes the wrong direction. For example, if the user has misunderstood a statement. In this case, he/she usually aborts the test or does not go to the areas you are interested in.
• You have hardly any chance to control the way the user works. For example, he or she only comments on the design instead of telling about his or her expectations. Or he or she forgets to think out loud
• You have little control over how the test users complete the tasks. Do they do it carefully? Or do they watch TV while doing the test? Do they play with their children?

If you think that all this disqualifies unmoderated user tests, I can put your mind at rest: Fortunately there are some proven tricks on how to get good results despite the difficulties. You’ll get them in chapter three.

2. How do I deal with the limitations of moderated tests?

As a good moderator of a usability test you speak as little as possible. You let the test user work and try not to influence him or her. You just ask the person to think out loud. If a test user is silent next to you in the usability lab, you can usually see without any problems whether he or she is reading a text in a concentrated way. Whether the test user is thinking or searching for something on the page. In order for you to be able to see this during an online test, you need the transmission of your test user’s webcam as well as the screen transmission. Pay attention to what you use when choosing the system. You can find a useful overview of various tools here: Remote user tests: the favourite tools of UX researchers.

3. How do I deal with the fact that I cannot react without moderation?

Especially through careful preparation, you can significantly improve the quality of results in unmoderated tests. My top tips for good unmoderated online tests are:

3.1 Pay attention to good test users

What is a “good” test user? Someone who comes from the target group of the product you are testing. In addition, the person should have a few more qualities:

• Reliable, so that he/she handles the tasks as intended.
• Diligently, so that he/she can work on the tasks independently.
• Attentive, so that he/she reads and understands your instructions.
• Ability to express oneself reasonably well or to think out loud. So that he/she comments in the video what he/she does and that is understandable for you.
• Technically reasonably skilled so that he/she can record and upload video.

In the case of unmoderated tests it is therefore even more important to pay attention to carefully selected test users.

3.2 Think of good tasks

Ideally, you do not have to say much about the tasks in a usability test. Sometimes you have to give a little more context so that the test users know what is expected of them. Put yourself in the test user’s shoes as much as possible when writing the tasks. Make sure that the sequence of tasks is good: An easy one to start with, so that they become familiar with the system. Then, slowly, more and more difficult tasks and finally an easier one.

3.3 Write a top briefing

The interview guide or test guide for your usability lab tests is important. But if something is not quite right with it, you can easily make up for it by moderating the test. With unmoderated tests, on the other hand, you don’t have the possibility. If something is wrong with the guide, the test users will try the wrong thing. They are confused, demotivated or cannot solve the tasks. Therefore, with unmoderated tests you have to put twice as much effort into the test guide.

3.4 Always, always do a trial test

As for any application, the same applies to any unmoderated UX test: You should never start without having tested with the target group. Because all too easily an unfavourable wording, a wrong link or a misunderstanding can lead to the whole test falling apart. Therefore: always run a test with the real questionnaire with the real system with real users. If this is not possible, then at least have a colleague test it. But he or she should give real answers and try to follow your instructions exactly. Just clicking through quickly once is not enough.

3.5 Check first answers from test users

Once the test has started, you can sit back. But not for long. Once the first answers start rolling in, you should check them. Did the test users actually understand the tasks? Were they able to complete them successfully? Did they indicate meaningful and plausible things in the questionnaires? Were they able to upload the desired screenshots, pictures or videos and can you watch them? It is not uncommon for problems to occur with one of these topics. The earlier you recognise them, the less answers you have to take out of the evaluation later.

3.6 Recruit more participants

To compensate for the limitations of unmoderated tests, you should recruit more test users than usual. Because it will always happen that someone has had unexpected technical problems that are not related to the tested product. That someone didn’t put as much effort into testing as expected. That there was an unsuitable test user. Or that individual user comments were unclear.

3.7 Limitations for tests with mobile devices

If you want to test an app or if the web application or site is mainly used on mobile devices, then unfortunately it becomes a bit more difficult with the choice of test users. After all, it turns out time and again that many users have technical challenges here. In the simplest case, if you are using a website or online service, the problems of the test users are:

• Media break between the application through which they receive their instructions (video conferencing tool for moderated tests and survey tool for unmoderated tests) and the application itself. It has proven to be effective to ask users to use a PC for the instructions and to operate the test object on the mobile device. Otherwise they will have to switch back and forth, which is a source of errors and annoying.
• Split or record the screen of the mobile device. A very good written manual (online or PDF, better yet video with screen recording) helps users to set this up and test it.
• Save and transfer or upload the video file. You should also explain this well. And if possible be available in advance in case of technical problems.

What you cannot do in online tests with mobile devices is to record the test user’s fingers. When testing in the usability lab, you can see whether test users are desperate to type or repeatedly swipe. In order to make sure that the observers of the usability test are aware of this, many of them use a camera that is permanently installed on the smartphone. It then shows the screen of the device together with the fingers of the test person from above.

With such constructions you record how the test users interact with the smartphone during usability lab tests.

3.8 Screen recording from iPhone, Android & Tablet

Some screen recording solutions for smartphones visualize user gestures and typing with symbols during recording. This way you can at least roughly see what the users have done. Of course, this option must also be activated. Unfortunately, you cannot rely on it one hundred percent. And on iOS this is generally not possible because the operating system does not allow it.

Mobizen

Mobizen is the app used by most Android users – technically good, though support and the website take some getting used to and have limited usefulness. The test version places a watermark on top of the recording, a monthly licence currently costs $3.00.

In iOS, screen recording is possible innately. All you have to do is to activate the feature under Settings > Control Centre > Screen Recording. Then, you can launch the screen recording feature from Control Centre.

TeamViewer

For moderated online tests on mobile devices, most recommend the TeamViewer software. With it, you see everything the user sees – but unfortunately no camera image. Only specialised tools offer the screen and webcam view that the user must also install like TeamViewer. And all these solutions are not cheap.

Skype

A cheaper solution is for example Skype. But you have to decide again between “share screen” and “transfer webcam”.

3.9 Alternatives to tests with mobile devices

You can avoid the problems with mobile devices by letting the test users test on PCs instead of mobile devices. This is sometimes possible even if you actually want to test a mobile application. Especially mobile websites can be tested quite well by letting the test users work with a desktop browser. Then ask them to switch it to the view with which you can test the smartphone version (e.g. in Firefox under Menu > Web Developer > Responsive Design Mode). For unmoderated tests I prefer to put the pages into a prototype. This way I make sure that all users see the same thing and that there are no technical problems.

This way a mobile website can be tested on a desktop browser. In the Axure prototype, the site is inserted into an inline frame. This works with live websites, but also with screenshots/mockups.

3.10 Purposes/questions for online tests & efforts

In the following situations you should generally rely on online tests, even if budget and circumstances would allow classic tests in the usability lab:

• The context of use is crucial for the application, and the use takes place on the road. For example, it makes sense to test an app for orientation in the mountains with users who are actually in the mountains.
• If you cannot go to the place where the application is to be used. For example, if you have an app that is used in factories or laboratories with restricted access, then online testing is a way to still test under realistic conditions.
• If you want to test with people you can’t get to otherwise. Typical example: Decision makers higher up in the hierarchy rarely take the time to come to you in the usability lab. If you cannot or do not want to come to them, an online test is the best solution.

Overview of types of online usability tests

This table makes it clear: unmoderated tests require more preparation, but the execution itself is very uncomplicated. Therefore, they are especially useful if you want to test with a large number of users.

4. Summary

If you are not testing remotely, you may not be able to do any user research at all. So you should at least try the methods and you will see. In many cases, remote testing can bring you valuable insights that you would otherwise have to do without. And in some cases the methods fit so well that they are even better than classic tests in the usability lab.

Tips

If you follow the tips above, you will get valid and usable results. Remember this in particular:

• Don’t just take an unmoderated test without video just because it’s supposed to be the easiest. Also, don’t just take a moderated test with screen transmission and video, because that is the closest thing to a usability lab test.
• Choose the method carefully. Choose the one that best fits your current question. Adapt it to the limitations of the particular online test.
• Try to test on laptop or desktop devices. Testing on mobile devices is becoming increasingly complex – but sometimes unavoidable.
• Just as no product should be released without a usability test, you should never skip a test run/pre-test.
• Give the test users the time to prepare and create the technical requirements. Make sure they have support if they have problems.
• Give the test users time to prepare and get the technical requirements right. Make sure they have support if they have problems.
• Also make sure that possible observers and yourself do not have any technical problems during a moderated test.
• Check the results for unmoderated tests right at the beginning of the test.
• Ask the test users as few questions as possible. Use open questions (text fields) sparingly, but do not abandon them.
• Do a plausibility check for all answers.

If you pay attention to these tips, you will get the most out of the online tests. With these you can advance your application, no matter what the external circumstances are. And you expand your possibilities and become a better UX researcher. 

So come on, go ahead and test online!