Why AI Testing Should Start With the User, Not the Model
The Future of AI Testing Is User-Centric, Not LLM-Centric
For decades, software testing was relatively straightforward. Engineers wrote code, defined expected behavior, and verified that the software behaved as intended. If a login button worked, an API returned the correct response, or a database query produced the expected result, the test passed.
Large language models broke that paradigm. Instead of deterministic outputs, we now work with probabilistic systems whose behavior is shaped by prompts, context, memory, tools, and reasoning.
How We Test AI Today
The first generation of AI evaluation naturally focused on the model itself. Teams asked measurable, repeatable questions closely tied to the language model: Was the response factually correct? Did the model hallucinate? Was the retrieved information relevant? Did the agent choose the right tool?
Those metrics became the foundation of modern AI evaluation. They're also becoming increasingly insufficient.
The Question Has Changed
As AI systems evolve from chatbots into autonomous agents, the fundamental question has shifted. We're no longer evaluating whether a model can produce a good answer. We're evaluating whether an agent can successfully help a human accomplish a task.
That distinction sounds subtle, but it represents a profound change in how AI should be tested.
People don't experience AI one prompt at a time. They experience conversations. A customer trying to resolve an issue doesn't ask one perfectly phrased question and disappear. They clarify details, ask follow-ups, change direction, provide incomplete information, and occasionally misunderstand the agent entirely. The interaction unfolds over multiple turns, with each response shaping what comes next.
Yet many evaluation frameworks still assess these systems as though every response exists in isolation.
When Every Response Passes but the Agent Fails
Imagine an AI assistant handling employee access requests.
Every individual response could score highly for relevance, factual accuracy, and groundedness. The retrieval pipeline may work perfectly. The tools may all execute successfully. The language may be clear, concise, and professional.
And yet the agent could still fail. It might grant access to someone who shouldn't have it, deny access to an employee who should, or send the user through an unnecessarily long approval process.
From the perspective of traditional LLM evaluation, the system performed well. From the perspective of the employee and the organization, it failed.
That gap is the challenge facing AI evaluation today. Measuring the quality of individual responses is no longer enough, because users don't judge agents response by response. They judge them by whether they solved the problem they came to solve.
Start With the User, Not the Prompt
A user-centric evaluation framework begins not with prompts but with people. Who is the user? What are they trying to accomplish? What information do they have? What constraints are they under? What would success actually look like from their perspective?
These questions force us to think beyond language generation and toward outcomes.
Consider two users asking the exact same question: "Can you give me access to this system?"
One is a new employee onboarding to the company. The other is an external contractor. The question is identical, but the correct response is not. The employee may be eligible for immediate access based on their role, while the contractor may require additional approvals, or may not be permitted access at all.
Evaluating whether the agent responded correctly requires understanding who the user is, what permissions they have, and the policies that apply to them. Without that context, it's impossible to know whether the agent made the right decision.
Accuracy, in other words, is no longer just about whether an answer is factually correct. It's about whether the answer is appropriate for a specific user in a specific situation. Context, identity, intent, and business rules all shape what "correct" actually means.
Test Workflows, Not Prompts
The same principle applies across customer support, healthcare, finance, education, and enterprise software. AI agents increasingly operate within workflows, not conversations alone. They retrieve information, execute tools, create documents, update databases, coordinate actions, and make decisions that extend far beyond generating text.
Testing these systems therefore means evaluating workflows rather than prompts.
This also changes how we define success. Traditional benchmarks optimize for response quality. Businesses optimize for outcomes: Did the employee gain the right level of access? Was the support issue resolved? Was the insurance claim processed correctly? Did the customer complete their purchase? Did the user accomplish what they set out to do?
These aren't language problems. They're user problems. An agent that produces eloquent responses but consistently fails to help users is ultimately less valuable than one whose responses are imperfect but reliably guide people to successful outcomes.
Evaluate the Whole Journey
As agents become more autonomous, another challenge emerges: evaluating a single response tells us very little about how an agent behaves over time. Did it recover after making a mistake? Did it ask appropriate clarifying questions? Did it maintain context throughout a long conversation? Did it know when to use a tool rather than keep generating text? Did it eventually achieve the user's objective?
These questions only become visible when the entire interaction, not the individual response, is the unit of evaluation.
This is where the industry is heading. Rather than grading model outputs, forward-looking teams are evaluating complete user journeys. They simulate realistic users with realistic goals, observe how conversations evolve over multiple turns, and measure success at the end of the interaction instead of after every response.
In many ways, this mirrors how humans evaluate one another. We rarely judge a colleague on a single sentence from a meeting. We judge them on whether they accomplished the objective. We care about outcomes, consistency, adaptability, and the ability to navigate complexity, not isolated moments.
AI agents deserve to be evaluated the same way.
The Bottom Line
The next generation of AI systems won't be distinguished solely by larger models or faster inference. They'll be distinguished by their ability to accomplish meaningful work on behalf of people. If that's the future of AI, our evaluation methods must evolve alongside it.
The future of AI testing isn't centered on the language model. It's centered on the user.
When we measure success through the lens of the people who rely on these systems, not merely the models that power them, we build AI that is not only more capable, but more useful.