Should You Actually Add AI To That? A Four-Question Audit For Product Teams — Smashing Magazine

Victor Yocco, PhD, is a UX Researcher with a focus on AI and the author of Designing Agentic AI Experiences (CRC Press, 2026), Design for the Mind (Manning, … More about Victor ↬
Weekly tips on front-end & UX.Trusted by 182,000+ folks.
A product team at a large health insurer wants to add AI that writes the full written determination for benefits appeals specialists. The technology works as specified. However, the specialists stop using it, and the ones who keep using it take longer on each case than the ones who ignore it.
A short evaluation with the specialists surfaces the problem fast: reading an AI-written determination carefully enough to sign it takes about as long as writing one, and the specialist also has to check for mistakes they would never have made themselves. The feature adds work to a task these people had already mastered.
Rescoping the release fixes it. A second version surfaces the policy provisions and prior decisions relevant to each appeal and leaves the writing to the specialist. The search step disappears, review time drops, and the part of the job the specialist is good at stays where it was.
The technology is identical in both versions, and the difference is which part of the task the AI handles.
Every interface change costs the user something before it pays anything back. They have to rebuild their picture of how the tool works, learn where things are now, and decide how far to trust what the AI hands them. Nelson Cowan’s review of working memory research (2001) put the limit at about four items held in mind at once. All three of those jobs draw on the same small supply.
An AI feature in an enterprise software setting requires more from the user than a redesigned menu, because it changes what they do all day. Hand the synthesis to an AI and the user becomes a checker. Give it the recommendation too, and the job shrinks to signing off. Either change costs user attention, and for people whose training is their standing at work, either one can feel like a demotion. For example, in one study, Sophia Ackerhans (2025) and colleagues found that physicians asked to sign AI-assisted decisions reported a more serious threat to their professional identity, and explanations of the AI’s reasoning raised both trust and that threat together.
We can call that the break-even line. Returns to users come in three forms:
For example, an AI-powered fraud screening tool shows the errors-caught form at work, since the analyst reviews flags they would never have raised themselves, which costs attention, and the catch rate pays for it. That puts the feature above the break-even line.
Four questions can help your team locate the break-even line before design work starts. Each one produces a design condition the brief can act on.
These questions do not replace the build decision. Funding, strategy, and existing commitments all weigh on that, and four questions cannot see any of it.
Run through all four questions together, since no single question settles the issue alone. Do it while the feature is still a proposal, before anyone sinks weeks into design. You will also need at least one person who does the task for a living, such as a loan officer, analyst, or benefits specialist. They do not have to sit in the roadmap meeting, and a short conversation tells you where the mental effort actually sits and whether the feature would help or add work.
Meet Smashing Workshops on front-end, design & UX, with practical takeaways, live sessions, video recordings and a friendly Q&A. With Brad Frost, Stéph Walter and so many others.
Count everything the task asks of the user: the thinking, the clicking, and the checking they will owe once AI is involved. Fewer clicks proves nothing on its own, because a feature can cut clicks and still leave the person working harder. Write down the cost even when it goes up, since Question 2 may find a return that pays for it.
A loan officer reading 40 financial disclosures to find the three that conflict with lending policy is thinking hard, and AI that surfaces those conflicts cuts that demand directly. Compare that with typing a loan number into a tracking system, which costs a few seconds and almost no thought. Pre-filling that field relocates the cost instead of removing it, because the officer now checks what the machine entered. Conventional wisdom says to automate the repetitive and monotonous tasks, and the pre-filled field is where that advice misfires. In this example:
You can find the tasks worth automating by comparing total effort before and after. Count the checking the user inherits on the after side.
Who the user is matters as much as what the task is. One feature can clear the line for a junior analyst and fall short for a senior one. Hand a junior analyst an AI-drafted risk summary, and they gain a starting point they lack the pattern library to produce on their own. A senior analyst, however, reads that same summary against the analysis already formed in their head, spends four minutes doing it, and learns nothing.
A finding like that tells you who to ship to. Scope the addition to the group it serves.
Compare the AI approach against the user’s current method, including the spreadsheets and workarounds they built for themselves. Genuine ambiguity is one reason AI comes out ahead when someone explores options, produces a starting point, or looks for patterns too numerous to spot by hand. Scale, speed, and consistency across a group of people doing the same job are three more. A fourth is catching what the current method reliably misses, which is how the fraud screening tool earns its place. The advantage disappears once the user has a practiced route to a known answer.
For example, an HR team building a succession planning tool finds the answer running both ways inside one product. Using AI to surface internal candidates whose skill combinations nobody would think to search for wins on ambiguity and scale together, since no single right answer ex