Artificial Intelligence, today, has led to a similar shift, if not an even more transformative one. In fact, UX is one of the industries that has led AI adoption in recent years and it shows. A Nielsen Norman Group (NN/g) study that analysed one million conversations with Claude.ai concluded that UX professionals, who represent less than 0.01% of the US workforce, generate a remarkable number of AI conversations. Specifically, out of the one million conversations, 7.5% were generated by UX professionals. Some of the more obvious uses of AI in research include converting audio interviews into transcripts within seconds, as well as assisting in tailoring consent forms, test plans and screener questions to different participant segments. The application extends further. Reinforcement learning algorithms have the ability to modify survey flows in real time, reordering or rephrasing questions based on a participant’s engagement level, hesitation or prior answers. Additionally, Natural Language Processing (NLP) can identify vague, contradictory or off-topic responses as they happen, asking participants to elaborate or clarify their input even before the session ends. As AI becomes embedded in research workflows, a new challenge emerges: determining where automation genuinely improves research and where it risks weakening the human insight that credible research depends on.

AI in UX Research: A snapshot

In the study “Revolutionizing Survey Data Collection with AI Powered Automation in Sample Selection and Response Quality,” Akintola concludes that AI-enhanced survey systems achieved a staggering 94.2% completion rate, compared to 78.6% in the control group. For UX researchers, whose core lies in identifying the fine nuances in every participant’s responses, AI also proves to be particularly helpful in diversifying linguistic diversity, with a Type-Token Ratio of 0.72 compared to 0.58 in traditional surveys, according to the same study. 

Specialized AI agents can analyze hundreds of pages of transcripts and screenshots to generate detailed, multi-page reports based on specific research objectives and discussion guides. This cuts time drastically, especially that some of these agents can do the job quite well, while others fall into the usual AI trap of filling gaps with fabricated or unsupported information.

The Limits of Automated Insight

When it comes to research participants, the story completely changes. AI can assist in predictive sampling whereby machine learning models analyze historical behavioral and demographic data to identify and recruit high-quality respondents who are most likely to provide consistent and complete data. These tools can also support the initial stages of defining user characteristics and mapping expected user journeys to better understand the research context. Nonetheless, this cannot replace real human participants. AI is rapidly improving the analytical layer of UX research, yet it does not resolve the most fundamental determinant of insight quality: who researchers are learning from. 

Despite technological progress, four persistent challenges continue to shape the credibility of research insights: the layered nature of human behaviour that extends beyond observable data, the growing reliance on self-service research practices that can erode study quality, the risk of AI amplifying weak participant input into seemingly authoritative conclusions, and the limitations of simulated users when compared with real human behaviour.

The value of research lies in the ability to observe subjects up close, understanding their behaviours, patterns and pain points and drawing conclusions that later set the path to improving a service or a product. This ability can be compromised when research functions rely heavily on AI-driven interpretation. There are subtle human nuances that AI tools cannot truly detect in the first place, let alone decode and decipher. Keywords like “disappointed” or “satisfactory” may be captured accurately by automated systems but it takes human involvement and interaction to truly grasp what such blanket terms might signify, especially when body language reveals a meaning separate from the verbal response. These nuances are often a gold mine for a researcher. That is why it takes real human participants to generate meaningful insight and seasoned experienced researchers to interpret that insight in whichever form it appears.

Interestingly, however, Gale M. Lucas et al., 2017 found that military personnel reported more PTSD symptoms to virtual humans than to anonymous surveys and Tamir Mendel et al., 2024 found that people disclose health information more readily to AI when controlling for trust. This is not a contradiction as much as an additional insight into how disclosure behaviour varies depending on context. When participant sample quality is strong, it becomes easier to adopt AI in appropriate stages of the research process. As such, it remains the researcher’s responsibility to understand the objectives and purpose of the study in order to tailor the research process accordingly. Intimate topics may justify the use of AI in some stages, while other moments may require direct human moderation. This also depends on the nature of the respondents and how comfortable they are sharing with a human versus a machine. None of this is new territory for researchers. They have had to exercise this judgement for decades. AI simply introduces new tools and opportunities into an already familiar process.

The Rise of Self-Service Research

Self-service research has democratised UX research and allowed more people to conduct studies easily by quickly setting up flows online and asking participants to respond remotely as well. This has added a valuable validation layer for many designers, marketers and product managers. Still, many of these remote, self-service platforms (and the studies conducted through them) are not designed or moderated by professional researchers with formal training (Schirra et al., 2023). This leads to a reduced focus on methodological rigour and research quality.

When a study is conducted without clear methodological foundations, the sample selection quickly loses credibility. Moreover, the untrained eye might easily miss broken links, faulty prototypes or confusing instructions. This poses a greater threat than a simple setup weakness. Instead, it could directly impact the way participants respond. Glitches lead to frustration and repetitive or unclear questions reduce response accuracy. Even from an ethical standpoint, Schirra et al. argue that skipping the quality assurance step causes respondents to perform unpaid labour, as they attempt to navigate these inconsistencies. In short, a faulty research setup paired with low-quality participant input leads to inaccurate outcomes. This is particularly dangerous because these outcomes do not exist in a vacuum. The assumptions made based on such outcomes are often scaled across teams and organisations. It is no secret to a seasoned researcher that erroneous conclusions lead to solutions disconnected from the real problems at hand.

AI Amplification Risk

Building on the notion that erroneous conclusions lead to disconnected solutions, one major watch-out here is that AI tools can fill gaps so convincingly that they may go completely unnoticed. For instance, no matter how high-level a generated report seems to be, a sceptical researcher should maintain a healthy level of doubt. Most available tools today are optimised for confident outputs. Rather than admit that information is missing, these generative tools may compensate with fabricated or unsupported details. Here as well, Akintola asserts that such modules seek to optimise three often conflicting goals: maximising completion rates, increasing the richness of responses and minimising cognitive fatigue. While the latter is crucial during the process to keep participants engaged, the former two are what make the final report look deceptively complete. 

Ironically, AI has become so good at filling the gaps it perceives, yet it does a poor job of detecting gaps in general. Researchers need to be aware of that and make sure they either have all facets of a problem properly covered in their prompts and study design. Otherwise, they should utilise AI’s own capabilities by explicitly directing it to look for potential blind spots. In a widely shared openAI video demonstrating the launch of GPT-4, the tool was asked to examine a room and the subject talking to it. Later during the conversation, another person entered the frame making bunny ears. This was not detected by the machine until it was explicitly asked about it. In short, UX researchers are better off asking AI to watch out for any bunny ears that might pop up in the responses.

Synthetic Users and the Limits of Simulation

Another major AI development, perhaps the most provocative frontier in UX research, is the emergence of “digital twins” and synthetic users. In theory, these systems are set up in such a way that allows them to simulate human behaviour, attitudes and responses at scale without having to recruit and compensate real participants and the logistical hurdles that come along. This seems enticing at first glance, and it’s not without merit. Unfortunately, these systems still cannot replace human participants. They are more of a derivative layer that depends on real-world data. In other words, in order to replace humans with quality synthetic users, researchers first need real-world data.

To put this into context, a 2024 Stanford study (Park et al., 2024) in which researchers built digital twins capable of replicating humans in survey responses resulted in responses with an accuracy of approximately 85%. The number seems convincing enough, demonstrating that synthetic users have an impressive ability to emulate real users. The caveat, though, is that they only managed to reach such levels of accuracy by building these synthetic users around rich, detailed data collected from real participants over a substantial amount of time: intensive two-hour qualitative interviews with more than 1,000 participants. When attempting to generate synthetic users using only brief persona descriptions or general demographic attributes, the accuracy dropped noticeably. A clear understanding of real users is a necessary precursor to any successful synthetic users.

However convincing they may appear, synthetic users lead to a lack of diversity in responses. Synthetic datasets often resemble human data on the surface but they fall short when it comes to linguistic variability and content diversity compared with real human responses. A whitepaper conducted in 2023 (Hämäläinen) compared real human accounts of video game art to GPT-3 generated outputs. The results were striking. Human respondents provided a wide range of titles. Conversely, AI mentioned a single popular game, Journey, 151 times, while humans only mentioned it 7 times. This is referred to as “journey bias,” whereby AI gravitates toward a single widely recognised example at the expense of the organic variety of references found in real populations with different backgrounds and cultures.

Even linguistically this tendency can be observed. A synthetic dataset can be five times larger than a human one, yet its unique word count can still be 2,000 words lower (Meyer, 2022). By design, AI tools use semantically dense words in order to appear coherent and articulate. Gravitating towards this tone and style regularly results in language that appears coherent but lacks the naturality of human dialogue. Filler expression, conversational irregularities, messiness and unpredictability and the value they add are, therefore, removed from the resulting data pool.

Language models are fed large volumes of human data and are trained by consolidating patterns across this data. This gives them access to a plethora of experiences and viewpoints. Synthetic users cannot, though, fully represent lived experiences. That is mainly because they can only infer general patterns and behaviours but when it comes to mirroring a single person’s lived experience, the technology falls short. To illustrate, when generating synthetic cohorts for health coaching scenarios, baseline models built exclusively from demographic information largely highlighted behaviours such as reflective motivation failing to represent other behaviours such as social and physical opportunities (Yun, 2025). This gets even trickier in personality-driven simulations where users tend to behave in “unpredictable” ways that current models cannot detect reliably.

Despite these limitations, synthetic users can still provide meaningful value when used appropriately. One tactical use would be during early or technical stages of research where a team is pilot testing prototypes, debugging analysis pipelines or performing “smoke tests” before running the study with real participants. The fast and efficient nature of these synthetic users makes them a useful preemptive step that can help avoid many issues that may arise during an actual study.

In short, synthetic users are far better employed as a complement rather than a substitute for real human participants and the depth they bring. Synthetic AI users are a significant milestone in the industry but their effectiveness still relies on substantial preliminary research. The purpose of in-depth UX research is to uncover the fine nuances in human behaviour. These aren’t obvious patterns, rather hidden deep within their cognitive and behavioural workings. That is why real participants remain crucial.

Conclusion

As artificial intelligence becomes increasingly present in research workflows, UX researchers are faced with a new responsibility: learning where automation genuinely strengthens research and where it risks weakening it. AI can dramatically accelerate tasks such as transcription, analysis and data processing. What it still can’t do is grasp and emulate human nuance, methodological rigour, participant quality and contextual understanding.

Throughout this paper, several risks illustrate this tension: the limits of automated interpretation, the methodological erosion introduced by self-service research, the amplification of weak participant input through confident AI outputs and the growing appeal of synthetic users that ultimately depend on real-world data.

Used strategically, AI can strengthen research workflows. But the foundation of credible insight remains unchanged. Real human participants are still the source of the understanding researchers seek to uncover.

References

Akintola, A. F., & Akanji, A. R. (2025). Revolutionizing survey data collection with AI powered automation in sample selection and response quality. International Journal of Advanced Statistics and Probability, 12(1), 40-51. https://doi.org/10.14419/8m5jht86

Hämäläinen, P., Tavast, M., & Kunnari, A. (2023). Evaluating large language models in generating synthetic HCI research data: A whitepaper. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23) (Article 237, pp. 1-19). ACM. https://doi.org/10.1145/3544548.3580688

Lucas, G. M., Rizzo, A., Gratch, J., Scherer, S., Stratou, G., Boberg, J., & Morency, L.-P. (2017). Reporting mental health symptoms: Breaking down barriers to care with virtual human interviewers. Frontiers in Robotics and AI, 4, 51. https://doi.org/10.3389/frobt.2017.00051

Ma, C., Xu, Z., Ren, Y., Hettiachchi, D., & Chan, J. (2025). PUB: An LLM-enhanced personality-driven user behaviour simulator for recommender system evaluation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25) (pp. 1-5). ACM. https://doi.org/10.1145/3726302.3730238

Mendel, T., Nov, O., & Wiesenfeld, B. (2024). Advice from a doctor or AI? Understanding willingness to disclose information through remote patient monitoring to receive health advice. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW2), Article 386. https://doi.org/10.1145/3686925

Meyer, S., Elsweiler, D., Ludwig, B., Fernández-Pichel, M., & Losada, D. E. (2022). Do we still need human assessors? Prompt-based GPT-3 user simulation in conversational AI. In 4th Conference on Conversational User Interfaces (CUI 2022) (pp. 1-6). ACM. https://doi.org/10.1145/3543829.3544529

Yun, T., Yang, E., Safdari, M., Lee, J. H., Kumar, V. V., Mahdavi, S. S., Aharony, R., Michaelides, A., Schneider, L., Galatzer-Levy, I., Jia, Y., Canny, J., Gretton, A., & Matarić, M. (2025). Sleepless nights, sugary days: Creating synthetic users with health conditions for realistic coaching agent interactions. Preprint.

Zhang, Y., Atiq, A., & Chow, W. (2024). Exploring the role of AI in UX research: Challenges, opportunities, and educational implications. In T. Cochrane, V. Narayan, E. Bone, C. Deneen, M. Saligari, K. Tregloan, & R. Vanderburg (Eds.), Navigating the Terrain: Emerging Frontiers in Learning Spaces, Pedagogies, and Technologies. Proceedings ASCILITE 2024 (pp. 556-560). ASCILITE. https://doi.org/10.14742/apubs.2024.1341

 

 

This whitepaper was co-written with Charbel Geagea, Copywriter for TestingTime and originally published for Merlien Institute in May 2026.