AI Chatbots Discourage Error Checking - NN/G

Introducing NN/G Membership | Access all self-paced courses for $249 per year | Join by Jan. 26, 2027 |
Find the training option that fits your learning style.
May 16, 2025 2025-05-16
Large language models (LLMs) are being widely introduced into professional workflows through both new standalone software and integrations with existing tools. One of the most important ways that these tools increase users’ productivity is through text generation. But generated content is prone to hallucinations, with the AI extrapolating outside of its training data to return outputs that are truth-like, but incorrect.
Designers of generative AI (genAI) products must help users identify and correct these errors. Most genAI tools currently fail in this responsibility.
When humans produce text, they alternate between writing and editing. GenAI tools make people more productive by accelerating the pace at which they can create content — the AI will near-instantaneously generate some the text based on the prompt, and the user needs to spend time only on editing.
Since LLMs are trained on a corpus of grammatically and syntactically correct content, their outputs may not need much copyediting. But for professional writing, editing also includes fact-checking and ensuring that any argument follows from the facts.
This work is not an afterthought — it is a difficult and time-consuming process. Users of genAI tools have been struggling to error-check their outputs successfully. Lawyers have been caught citing cases that don’t exist. Scientists refer to hallucinated papers and journals. Doctors end up trusting erroneous AI diagnoses over their own expertise.
It is irresponsible to blame the users for misinformation generated by LLMs. People are efficient (not lazy). Users adopt genAI tools precisely because they come with the promise of greater efficiency. They will not go out of their way to check the work if the effort of doing so does not seem proportional to the AI’s likelihood of being wrong.
It’s true that an exceptionally careful user can reduce the number of mistakes the LLM will produce. But doing so requires additional knowledge and user effort.
To verify the answers provided by a chatbot, the user will need to follow the same steps as they would when reviewing any other text, and then some.
Some of these steps may be shortened with careful prompt engineering. For example, asking the LLM to generate one small and digestible part of a text at a time allows users greater control over what is written and how it is incorporated into the larger whole of the work. But these strategies are workarounds that require user effort and an in-depth understanding of how AIs hallucinate. And working through a document one part at a time is possible only for expert users who already know what those parts ought to be.
Humans do not check every single novel piece of information for accuracy; the effort to do so would be unjustifiable. The extent to which we take a claim at its value is based on how closely it fits with what we already know and the trustworthiness of the source.
Outputs of genAI tools mimic certain attributes that we associate with authoritative sources. Their tone is unerringly confident, regardless of the accuracy of the response. They are grammatically correct and meticulously formatted. Through a phenomenon called the halo effect, users’ positive perception of one attribute of the LLM’s responses causes them to have a positive predisposition about all its other attributes — including accuracy.
While we ought to consider AI-generated content an early draft, due to these signals we often treat it as finished work. Users frequently have such confidence in an LLM output that they do not even read through it end to end. For example, in our recent round of intern hiring, we were disappointed to see cover letters ending in “Let me know if you need anything else!”
Additionally, nothing about the chat interface (a plain text box) acknowledges that the LLM’s outputs might not be entirely accurate outside of a small label: XYZ bot can make mistakes, please check responses. Instead, chatbots typically encourage users to move on from the response by prompting them with a new question.
Improving writing productivity is not the only application of genAI. These tools are often used as virtual experts, providing advice or performing tasks outside of the user’s field of expertise. In practices such as vibe coding, where an AI generates code based on plain-language prompts, the user does not need to know anything about software development to create a usable program.
A side effect of this mode of usage is that users lack a sufficient understanding of the subject matter to meaningfully validate the LLM’s outputs. A vibe coder has no way of evaluating the quality of the code for security vulnerabilities, and a guidebook author might find out they gave bad advice only when customers start getting poisoned.
But far from all AI-generated answers are hallucinations. As genAI performance progresses to a point where the majority of LLM responses are correct, even users who started out by verifying every statement may start to feel that their effort is not paying off. As users increasingly take AI outputs at face value, they might lose the expertise necessary to evaluate the accuracy of those outputs. As a Microsoft paper by Hank Lee identifies, operators who come to rely on genAI for routine decisions end up being unprepared for situations that require their intervention.
While users can choose to collaborate with an LLM section by section to create a document or a working program, they can also just ask it to produce the entire text by itself. While this is a much faster way of getting an output, it creates additional problems when it comes to verifying the output’s quality and troubleshooting its constituent parts.
Safety-systems researcher David D. Woods observed that the task of analyzing a whole is made far more difficult with