ChatGPT writes screening questions

Table of content

1. Writing professional screening questions
1.1 Screener checklist
2. Case Study Mortgage Customer
3. Research the topic and profile with ChatGPT
4. Write Screening Questions with ChatGPT
4.1 Opening Question
4.2 Follow-up question regarding mortgage bank
4.3 Follow-up question regarding the timing of mortgage completion
4.4 Expert exclusion
5. Conclusion
6. Summary

1. Writing professional screening questions

Writing screening questions is a craft that needs to be learned. We have published a Screener Guideline E-Book for this purpose. By applying the checklist provided in it, you can already optimize a lot. Nevertheless, developing good screening questions requires a significant amount of work.

1.1 Screener checklist

For your reference, I pasted the full checklist here:

1. Setup screener
✅ Get yourself familiar with the topic/criteria
✅ Understand your exact test user profile
✅ Start with generic questions and then go to specific ones
✅ Reuse existing questions from your team if you can
✅ Only ask one information at a time – split questions if necessary
✅ Only write up to 8 questions If you look for different profiles, split the order if necessary

2. Choose question format
✅ You can choose between single-select and multi-select
✅ Use open questions to double check critical information

3. Question guidelines
✅ Avoid asking for agreement, especially don‘t use yes/no questions – try to give two ends of a scale instead (e.g. instead of “How much do you like […]?”, ask “How would you rate […]?”)
✅ Make sure your question is short and easy to understand (short sentences, simple words, explain expert terms…)
✅ On the other hand, your question needs to be specific and leave no room for interpretation (avoid specifying terms like „active user“, ”regularly”,…)
✅ Avoid “loaded questions” which point the user in one direction

4. Answer guidelines
✅ Make your questions visually appealing (use consistently capital letter at the beginning and “.” )
✅ Make sure you use gender-inclusive language
✅ When you need to come up with alternative answers, make some research about what other options are relevant to make the question reusable
✅ When you give options with numbers (e.g. frequencies: 1-2 times per week / 3-5 times per week / …), give a wide range and make sure there is no gap or overlap between steps
✅ Everyone needs to be able to answer the question – Think about including an extra option „Other“ / „None of the above“ / „I am not sure“ / „I prefer not to answer“
✅ Vary the position of the required criterion (don‘t always place it on top)

2. Case Study Mortgage Customer

I would like to use a typical screening example from our banking customers. It revolves around individuals who recently obtained a new mortgage.

Test user profile criteria:

• Individuals who have obtained a new mortgage for a home in Switzerland within the last 6 months.
• Mix of UBS and non-UBS mortgage customers.
• Exclusion of individuals working in the banking and finance industry, UX and design (including marketing, user research, market research, innovation management).

Typically, a target group like this is screened with approximately 5 screening questions. In this article, I will generate these questions using ChatGPT.

3. Research the topic and profile with ChatGPT

To familiarize with the topic and user profile, ChatGPT can be a valuable tool. The only restriction is that the inquiry is limited to knowledge up until September 2021. To leverage up-to-date information, it is recommended to switch to ChatGPT Plus.

Firstly, I would like to find out which banks offer mortgages in Switzerland.

Furthermore, I would like to understand which typical banking products are offered to retail customers in Switzerland.

With just two simple questions, I could learn more about the banking landscape in Switzerland and the financial products offered to retail customers. I am ready to have ChatGPT generate the first screening question for me.

4. Write Screening Questions with ChatGPT

4.1 Opening Question

The first generated version is already quite good, but I would like to make some further optimizations. I will prompt ChatGPT to adjust, remove, or add some options. A good screening question should also include a trap answer option to filter out “quick responders” or “liars.” Since ChatGPT doesn’t understand the concept of trap questions, I will directly instruct it to add “SpaceX bonds” as such a fake answer option.

The question now meets all the recommendations from the checklist. Assuming that the screener logic only allows participants who have selected at least “Option D – Mortgage,” I can directly address survey about the mortgage in the follow-up question.

4.2 Follow-up question regarding mortgage bank

I want to know which bank the mortgage was taken out with. This time, I deliberately prompt to use a single-select format.

Since the banking landscape in Switzerland is diverse, I would like to add four additional options. I choose not to include another trap answer option at this stage. The last option “I did not take out a mortgage” is recommended to be included in subsequent questions to correct any false information provided in the previous question.

4.3 Follow-up question regarding the timing of mortgage completion

To finalize the mortgage screening, I would like to generate a question that inquires about the timing of mortgage completion. Ideally, I want individuals who have completed a mortgage within the last 6 months. However, due to the potential shortage of candidates, I will also ask about broader timeframes without using a screen out.

4.4 Expert exclusion

As the final set of questions, I want to exclude experts and industry insiders. These are typical questions that are already present in our question library on the order form at TestingTime. However, I still want to demonstrate how quickly the question can be generated. To simplify the complexity, I will first ask about the industry and then move on to the specific role.

I would like to optimize the generated result. I want to add three more industries, in particular add market research as an industry.

Lastly, I would ask about all specific professions in a relatively closed manner.

5. Conclusion

All screening questions for this case study could be generated using ChatGPT. Good prompting makes a difference. Initially, I needed several attempts to achieve the desired result, but I quickly learned how to instruct ChatGPT more precisely and effectively. The AI tool is also very good at learning the context and preferences over time. It remembers, without specific prompting, to add small things like “Other” or “None of the above” as answer options. ChatGPT truly saves a lot of time when it comes to listing various answer options, such as financial products or banks. These are typically things that would require tedious manual research.

I am impressed by ChatGPT and believe it can also be used for generating regular online surveys. I also see great potential for its application in analyzing research results, especially free-text data. We are also looking into the possibility to directly embed it into the TestingTime order form on the Screening step. In a ideal world, a customer can describe the criteria similar as in Chapter 2 and with a single button click generate all these questions automatically. #neverstopdreaming 🙂

6. Summary

ChatGPT won’t replace you as a UX researcher, but it can definitely make your future work more efficient, whether you’re involved in planning, conducting, or analyzing user research studies. Particularly in the area of screening questions, ChatGPT has shown tremendous support.

A good summary? This was entirely generated using ChatGPT. Also the title image was generated using DALL-E2.