One AI Output Is an Example, Not an Evaluation - NN/G

Find the training option that fits your learning style.
August 14, 2026 2026-08-14
Suppose you ask an AI customer-service system the same question several times: “Can I return an opened product after 30 days?” One answer may explain the policy accurately. Another may omit an important exception. A third may confidently promise a refund that the customer is not entitled to receive. Which of these answers represents the system? The answer is: all of them, taken together — and none of them, taken in isolation.
Yet teams often evaluate AI systems by running them once, inspecting the result, and drawing conclusions about what AI can do or how it should be used. This is not because they are lazy or careless. It is because decades of deterministic software have taught us that a feature that works once will work the same every time. AI systems offer no such guarantee. One good output demonstrates that a system can perform a task. It does not show how often or how reliably the system will do so.
A nondeterministic system can produce different outputs when given the same input.
Language models assign probabilities to possible next tokens and select among them as they generate a response. As a result, submitting the same input repeatedly may produce answers that differ in wording and quality.
This variation can be substantial. Research comparing repeated generations from the same language models has found meaningful differences in performance across runs and has shown that evaluation results can depend on how outputs are generated.
A single output is an example, not an evaluation.
Imagine that we want to evaluate the usability of an ecommerce checkout flow quantitatively, perhaps for a UX-benchmarking study. We would not ask one participant to complete one checkout task and conclude that the site has perfect usability simply because that participant succeeded.
Instead, we would define a set of representative checkout tasks that involve different types of items to be purchased. We would then observe many participants as they attempted them and would report averages for metrics such as task success and time on task. We would also report confidence intervals to indicate the uncertainty around those estimates ––– how much those estimates are likely to vary across the broader population of users.
An AI evaluation follows similar logic. Instead of asking people to complete tasks with an interface, we ask an AI system to produce outputs for a set of test inputs.
Quantitative UX Study
The system being evaluated
The interface or product
The model, prompt, settings, tools, and supporting data
The range of situations tested
Representative user tasks
Representative test inputs
One data point (i.e., observation)
One participant attempting one task
One AI output corresponding to one test input
Task success, time on task, or error rate
Accuracy, task success, or a quality score
Sources of data variability
Differences across tasks and participants
Differences across test inputs and repeated runs
The analogy is not exact. An AI run is not a human participant. However, both types of evaluation rely on multiple observations to estimate how the system performs.
Quantitative-UX researchers summarize results across tasks and participants, rather than relying on one successful attempt. AI evaluations should do the same: use representative inputs, collect enough observations, and report average performance, together with variability and uncertainty.
We do not need to invent a new science of AI measurement; we need to remember familiar principles of experimental design and statistics.
Researching AI or anything else? This live online course teaches you how to use AI tools to speed up your research flow.
Suppose that we want to evaluate an AI system that answers customer-service questions using a company’s policies.
For illustration, we could select 10 representative questions covering different topics, levels of complexity, and types of customer situations. For example, the questions might address returns, cancellations, damaged orders, subscriptions, and warranties. We could then submit each question to the system 5 times, producing 50 answers. (These numbers are not universal recommendations. A real evaluation may require more questions or more runs, depending on the variability of the system and the importance of the decision. Choosing the number of questions and runs is a topic that we don’t cover in this article.)
Before conducting the evaluation, we would define what constitutes an acceptable answer. For example, an answer may count as acceptable only if it:
An answer that fails any of these would be considered unacceptable. (Note that this particular output-quality metric happens to be binary, but more complex metrics could be defined.)
We could then calculate each question’s percentage of acceptable answers across its repeated runs and then summarize performance across all 10 questions.
At minimum, we would report:
A confidence interval represents a range of plausible values for the system’s average performance. It reminds us that the score obtained in one evaluation is an estimate, not an exact and permanent property of the system.
Again, the purpose of the example is not to prescribe 10 questions and 5 runs as universal numbers. It is to illustrate the basic design: test multiple representative inputs, run each input repeatedly, and summarize the resulting distribution of scores.
Note that any evaluation is a snapshot of a particular system at a particular time. Like with UX-benchmarking studies, you need to carefully document all the details: record the model and version, prompts or instructions, settings, tools, context, and evaluation date. If any of these components change, the evaluation may need to be repeated.
The example above helps us answer two separate questions: