Synthetic Users: If, When, and How to Use AI-Generated “Research” - NN/G

Introducing NN/G Membership | Access all self-paced courses for $249 per year | Join by Jan. 26, 2027 |
Find the training option that fits your learning style.
June 21, 2024 2024-06-21
Of the new AI-based tools flooding the landscape of user and product research, few have caused as much debate as AI-generated users — also known as “synthetic users”.
A synthetic user is an AI-generated profile that attempts to mimic a user group, providing artificial research findings produced without studying real users. A synthetic user will express simulated thoughts, needs, and experiences.
A synthetic user is based on large language models (LLMs) that have been trained on vast amounts of data about people.
While general-purpose AI chatbots like ChatGPT can be used to create synthetic users, there are also dedicated AI-based platforms designed for this task — perhaps the most famous being the company named Synthetic Users. (In this article, we use capitalization to refer to this company’s product and lowercase to refer to the concept of synthetic users.)
In a conversation with Hugo Alves, cofounder of Synthetic Users, we learned about the product's practical applications. According to Alves, Synthetic Users is particularly useful in scenarios where swift decision making is crucial and absolute certainty isn't required. The tool allows teams to access the thoughts, reflections, and emotions of specific user groups, providing timely and frequent access to user insights.
At first glance, a Synthetic Users’ synthetic user profile looks similar to a user persona—it contains demographic details like a name, age, and education level. Like a persona, such a profile is intended to be an imaginary representation of a real group of people. However, unlike a persona, the profile is “alive” — meaning that you can chat with it.
To generate these AI-generated users, you specify the user group and the goal of your study.
Medical-detailing representatives who work in Latin America
How do medical-detailing representatives work?
The product then uses some dynamic parameters to create different users and uses the study goals to craft interview questions. Within a few seconds, your synthetic users have been created, and interview transcripts are ready to read. Synthetic Users provides the ability to ask followup questions and continue the interview (which we tried).
We have long argued that UX without user research isn’t UX. But, we did not anticipate that people would attempt to conduct user research without real users. We need to rephrase our slogan:
UX without real-user research isn’t UX.
In our interview, Alves stressed that he does not intend for his product to replace user research completely.
“You're never gonna stop talking to real people and you shouldn't. You shouldn't. There are some decisions that you shouldn't even go to synthetic users. You should just go to humans.” Hugo Alves, cofounder of Synthetic Users
However, the current marketing copy on the Synthetic Users site does not clearly reflect that stance.
If you know NN/g, you know that we never recommend that teams skip user research. Research with real people provides many intrinsic benefits; for example, it creates empathy and builds a vivid representation of the user in each team member’s mind (something reading a transcript from a synthetic user does not adequately do).
Putting those benefits aside, we tested synthetic users to see if there was a qualitative difference in the insights gained from real users compared to synthetic ones.
We used Synthetic Users and ChatGPT to generate synthetic users and insights for three studies we had performed with real users. Currently, these tools are limited to replicating text-based attitudinal methods, specifically interviews and surveys. In our evaluation, we focused on interviews.
In Synthetic Users, we specified the user group and the goals for our research. The tool then generated fake people and performed simulated interviews for us.
Since ChatGPT doesn’t generate the same outputs as Synthetic Users without a lot of prompting, we began each synthetic user interview with a prompt that asked the chatbot to pretend to be a particular person.
Example prompt used with ChatGPT:
Pretend you are a medical detailing representative working in Bogotá, Colombia. For some context: You've been working in the role for 6 years. Your day-to-day work involves meeting with doctors in hospitals and clinics around Bogota. You use a CRM solution that helps you track and plan your doctor visits. Pretend you are a participant in a user research session. I will be the interviewer.
While ChatGPT and Synthetic Users produced similar content, speaking with a synthetic user generated by Synthetic Users felt a bit more like speaking to a person (although the responses were still very unhuman and wordy).
Our experimentation revealed some interesting possible use cases for synthetic users. However, we also found the results to be much less useful than many UX and product professionals claim.
When used responsibly by mature research teams, AI-generated user profiles can complement real user research.
One of the advantages of a synthetic user is that it can synthesize vast amounts of available data about a specific user group and present it in a digestible way.
For example, there are products and systems designed to support the work of medical-detailing representatives, academic literature regarding medical detailing, forums dedicated to medical-detailing representatives, and more. LLMs have been trained on all this data. When we asked a synthetic medical-detailing representative to describe a typical day, the result was very similar to what real users told us in our study.
Response from a synthetic user:
Sure, I'd be happy to. My day usually starts around 7 AM when I go over my schedule and prepare for the meetings I have lined up with doctors and healthcare professionals. First, I make sure I have all the necessary materials —brochures, s