Table of Contents

1. Phone number verification through SMS
2. State of the art screening
3. Profile prediction algorithm
4. Quality prediction algorithm
5. Reconfirmation of profile criteria and availability
6. Email and SMS reminder

1. Phone number verification through SMS

Initially, we only supported email verification during a test user signup. We collected the smartphone phone number, but didn’t verify whether the number was actually reachable. As a consequence, reminder SMS didn’t go through or even worse, a customer reached an invalid number when trying to reach out. To avoid such poor customer experience, we took the verification process to the next level and introduced a phone number activation using an SMS code. Without this activation, a test user will not be placed as a candidate. It also allows us to screen out fake users, double profiles or bots.

2. State of the art screening

Every test user recruitment comes with a screening, where we check if a test user fits the profile requested or not. A screening is composed of a collection of screening questions with built-in logic to screen in/out candidates. Writing good screener questions is a science. It must be as short as possible (to ensure a high conversion rate) but open enough to not convey what profile we are looking for. At TestingTime, we write the screener based on the requested profile criteria. We have built a best-practice screener question catalogue, with questions we know work well. Each answered screener question comes with an expiration date to ensure a candidate doesn’t need to reply to the same question every time, but still frequently enough to guarantee up-to-date profile information. TestingTime made screening its expertise our whole Customer Success Team is trained and focused on this.  

3. Profile prediction algorithm

In an ideal world, we would already know who matches the criteria for a specific study. A lot of our competitors ask an endless catalogue of questions when test users sign up. At TestingTime we decided to only ask as few questions as possible during the signup process and build their profile by asking them additional questions only when we have a potential study for the test user. Our goal is to send as few screening invites as possible to generate the necessary number of candidates. 

Too many invites to a screening with no tests tend to frustrate test users after a while. That’s why some potential test users don’t reply to all questions anymore. Therefore we have to make a wild guess or predict candidates’ answers. In order to solve this problem, we have built an algorithm to predict whether the person may fit the wanted criteria or not. Based on that we can prioritize who to ask first. This results in fewer invites and increases the willingness of test users to participate in screeners. 

test user quality

4. Quality prediction algorithm

Based on data our quality algorithm predicts whether a candidate will return a good or bad rating from the customer. This check is based on another machine learning algorithm which gets better with every rating we receive. That’s why it is very important for all customers to rate their participants at the end of a study. Once enough candidates for a study are generated, our algorithm picks the right mix and confirms their participation. At the moment, the algorithm is only used to support the selection. The final call is made by the responsible customer success manager. In the near future, we believe that the entire selection process can be automated. 

5. Reconfirmation of profile criteria and availability

The profile and the quality prediction algorithm screen out a lot of potential misfits and no-shows. However, we have learnt that human misunderstandings are still hard to detect. This can happen when a screener question is misunderstood or by mistake a wrong option was selected. Therefore we have built-in an extra layer of reconfirmation into the system. When the system asks a final candidate to reconfirm their availability, the person is also asked to reconfirm the required criteria. This last line of defence filters out potential misfits.

6. Email and SMS reminder 

Another thing which can go wrong is that a scheduled test user simply forgets about the test or drops sick. We send an email and SMS reminder to every test user. In both reminders, we make it super easy to cancel. We would rather know that the test user can’t make it and find a replacement than not knowing about it.

With all these checks in place, we managed to get our no-show and misfit rate to a stable low number and to ensure having no study junkies at our customers’ test. However, we don’t stop our efforts in improving TestingTime’s test users quality even further. For example, we plan to look into ID verification. With many online banks coming up, there are now great solutions to verify an ID online. We believe it could bring even more trust into the quality of a test user.