Synthetic Users vs. Human Respondents: Same Conclusion, Less Variation - NN/G

Introducing NN/G Membership | Access all self-paced courses for $249 per year | Join by Jan. 26, 2027 |
Find the training option that fits your learning style.
October 16, 2026 2026-10-16
More teams are turning to synthetic users (AI-generated stand-ins for research participants) to run studies without recruiting real people. We’ve previously explored how synthetic users perform in interviews. This time, we wanted to test them on a more scoped task: answering a single open-ended question from a survey.
To do this, we reused a question from our earlier survey on UX terminology, in which 604 tech and design professionals told us what name they’d use for UX work. We posed the same question to 120 synthetic respondents and then compared the two datasets. Both datasets led to the same overall conclusion, but the synthetic responses varied much less than the human ones.
We obtained the synthetic responses in February 2026 using Synthetic Users, a generative-AI-based platform that simulates research participants and their responses.
We created 120 synthetic respondents intended to represent tech and design professionals worldwide. We provided Synthetic Users with the research plan, which included the research questions, inclusion criteria, exclusion criteria, and survey questions. Synthetic Users automatically assigned each respondent a profile with characteristics such as age, location, profession, cultural background, organizational structure, and personality traits.
We requested that the synthetic respondents be asked the same open-ended question used in the original human-participant study:
Question: Which professional field or discipline do you believe best describes the following work?
An individual gathers insights about people’s needs and experiences and turns those insights into recommendations for how a product or service should be planned, built, and improved.
Please name only 1 field or discipline.
Two aspects of the platform were outside our control. First, Synthetic Users simulates interviews, so the synthetic respondents answered as if speaking to an interviewer and often gave longer answers than the humans’ survey responses. Second, the platform sometimes reworded the question — for example, asking about “people’s needs, behaviors, and experiences” rather than “people’s needs and experiences.” Such rewording is probably meant to mimic how human interviewers rephrase questions in conversation, but even small wording changes can prime respondents or shift how they interpret a question, so we cannot rule out that they influenced the synthetic responses.
These differences point to a broader limitation of synthetic-user platforms: researchers have limited control over how questions are presented and the presentation may be a confounding variable in the study.
We analyzed the synthetic and human responses the same way. Because some respondents named more than one field, we coded each response for the keywords it contained rather than forcing it into a single category.
Our live online course, Accelerating Research with AI, teaches you which AI tools work best to improve research speed without reducing quality.
Both datasets led to the same primary conclusion: no alternative term has displaced UX as the dominant name for this type of work.
Synthetic respondents were even more likely to use UX-related terminology than human respondents. This 14-percentage-point difference was statistically significant, but it would not change the practical conclusion of our original study: UX was clearly the dominant term in both datasets.
This difference is worth considering in context. Our human sample was recruited partly through our own channels, such as our Practitioner Panel. As we noted in the original study, the recruitment likely resulted in a sample that was especially familiar with UX terminology and that may thus overestimate how commonly the term “UX” is used among tech and design professionals.
The synthetic respondents were not recruited through these same channels and therefore did not have this particular sampling limitation. In fact, 67% of the human dataset consisted of UX professionals, compared with only 20% of the synthetic dataset. (This difference was by design: We established quotas for different professional roles when generating the synthetic sample to ensure representation across a broader range of professions. The final synthetic sample closely matched these quotas.)
In theory, the synthetic dataset offered an opportunity to represent a broader range of tech and design professionals. Yet, UX-related terminology was even more common in the synthetic data than in our already UX-familiar human sample.
We cannot determine from this comparison exactly why this happened. However, previous research on synthetic respondents has found that their answers can be more concentrated around common or expected responses, with less of the variation found among real people.
The human respondents used a much wider range of terminology than the synthetic respondents. Human participants used more than 50 distinct labels, including “consumer-behavior research,” “digital-experience strategy,” “journey management,” and “human-factors engineering.” Most of these alternatives appeared only once or, at most, a few times.
Synthetic respondents, by contrast, collectively produced a much narrower set of responses. The difference was most apparent in the less-common answers: synthetic respondents produced none of the unusual or specialized terms that appeared in the human data, such as “digital management” or “business analysis.”
This pattern resembles mode collapse — a machine-learning term describing a generative model that keeps producing a narrow set of outputs instead of the full range it was trained on. (The name borrows the statistical sense of “mode”: the most common value in a dataset.)
In synthetic-users research, mode collapse refers to synthetic data that reproduces the most common data point b