Updated October 2026
The demo lasted twelve minutes.
Every question worked.
“What’s our annual leave policy?”
The assistant answered correctly.
“Can contractors claim home-office expenses?”
Correct again.
“Where can I find the parental-leave policy?”
It returned the right document.
Someone in the meeting said what people often say during a successful AI demo:
“This is ready.”
It wasn’t.
Three days later, the same assistant was tested against a larger set of employee questions.
It confidently answered a policy question using an outdated document.
It cited the correct handbook while giving an answer that was not actually supported by that handbook.
It failed to find a policy because the employee used an abbreviation instead of the full term.
And when asked a question for which the company’s documents contained no answer, it invented one.
The model hadn’t suddenly become worse.
We had simply stopped testing it with questions we already knew it could answer.
That difference—between a convincing demo and a trustworthy system—is what AI evaluation is really about.
This article follows a fictional company through that process.
No leaderboard.
No “Model A scored 89.4 while Model B scored 88.9.”
Instead, we’re going to answer a more useful question:
How do you decide whether an AI system is actually good enough to release?
The System We’re Testing
Imagine a fictional company called CedarWorks.
It has 8,000 employees across several countries.
The company builds an internal AI assistant called Ask Cedar.
Employees can ask questions such as:
“How much annual leave do employees in India receive?”
“Can I work remotely from another country?”
“What documents are required for parental leave?”
“Who approves professional-development expenses?”
The architecture is straightforward.
An employee asks a question.
The system searches approved HR documents.
Relevant passages are retrieved.
Those passages are sent to a language model.
The model generates an answer and provides links to the supporting documents.
In other words, Ask Cedar is a RAG system: retrieval-augmented generation.
The development team initially evaluates it the same way many teams evaluate AI systems.
They ask it questions.
They read the answers.
The answers look good.
Everyone is happy.
That is not evaluation.
That is inspection.
And inspection is useful—but it is not enough.
Google Cloud describes production GenAI evaluation as moving away from simply looking at a few outputs toward repeatable measurements of qualities such as groundedness, instruction following, safety and answer relevance.
The challenge is deciding what should actually be measured.
Table of Contents
Before Metrics, Define What “Good” Means
The CedarWorks team initially makes a mistake.
They begin by asking:
“Which evaluation metric should we use?”
Accuracy?
Groundedness?
Semantic similarity?
An LLM judge?
Human ratings?
Those questions come too early.
Before choosing a metric, the team needs to define what the system is supposed to do.
For Ask Cedar, they write a release contract.
| Requirement | What success means |
|---|---|
| Answer correctness | The response accurately reflects approved HR policy |
| Grounding | Claims are supported by retrieved company documents |
| Retrieval | The system finds the right policy even when wording differs |
| Uncertainty | It admits when the documents do not contain an answer |
| Permissions | Employees cannot access documents they are not authorized to see |
| Citations | The cited document actually supports the answer |
| Usefulness | The employee can understand what to do next |
| Performance | Response time is acceptable for normal use |
| Cost | The workflow is economically sustainable at expected volume |
That table becomes much more important than the model leaderboard.
Because Ask Cedar’s job is not:
“Produce impressive text.”
Its job is:
Help employees get reliable policy answers without creating new risk.
That is the evaluation target.
Google’s agent-evaluation guidance makes the same broader point: start with an explicit definition of success for the specific application, because vague goals such as “the agent should be helpful” cannot be evaluated meaningfully.
The First Test Set Changes the Conversation
Instead of asking five demo questions, CedarWorks builds an initial evaluation set of 100 questions.
The questions do not all look alike.
Some are straightforward:
“How many annual leave days do full-time employees receive?”
Some use informal language:
“How many days off do I get?”
Some contain abbreviations:
“What’s the WFH rule for contractors?”
Some deliberately ask about information that does not exist:
“Does CedarWorks provide a company car to every engineering manager?”
Some contain conflicting context:
“The 2024 handbook says X, but the 2026 policy says Y. Which applies?”
Some test permissions:
“Show me the compensation policy for executives.”
Some are ambiguous:
“Can I work abroad?”
That last question sounds simple.
But the correct answer might depend on country, employment type, duration, tax rules and manager approval.
The point is not to make the AI fail.
The point is to discover where it fails before users discover it for you.
OpenAI’s current evaluation guidance similarly recommends building evaluation datasets that include normal cases, edge cases and adversarial cases, while continuously adding new examples discovered from real usage.
The Digital Stackroom Evaluation Loop
For production AI systems, I would use this cycle:
Test Set → Run → Measure → Diagnose → Improve → Regression Test → Release → Monitor
It is intentionally a loop rather than a one-time certification.
Test Set
Create representative examples of the work the system will perform.
Run
Execute the same evaluation against the current system configuration.
That means the actual prompt, model, retrieval configuration, tools and business rules—not an isolated model in a playground.
Measure
Score the results using the criteria that matter for the application.
Diagnose
Do not stop at:
“Accuracy dropped.”
Find out why.
Was the wrong document retrieved?
Did the model ignore the document?
Was the question ambiguous?
Did a tool fail?
Improve
Change the component responsible for the failure.
That could mean changing:
the prompt,
retrieval,
chunking,
metadata,
permissions,
tool descriptions,
workflow logic,
or sometimes the model itself.
Regression Test
Run the full evaluation set again.
A fix for one problem can introduce another.
Release
Promote the change only if the system meets the required thresholds.
Monitor
Collect production failures and turn useful ones into new evaluation cases.
NIST’s AI Risk Management Framework treats measurement as a lifecycle activity rather than a one-time exercise. Its Measure function calls for AI systems to be tested before deployment and regularly while operating, with metrics reassessed as systems, risks and context evolve.
That is the important idea.
An eval set is not something you finish.
It becomes part of the product.
Then Ask Cedar Scores 92%
The first run looks encouraging.
Ask Cedar passes 92 of the 100 tests.
Someone says:
“92% is pretty good.”
Maybe.
But that number hides something.
The eight failures are not equally important.
Five are minor.
For example, the answer is correct but unnecessarily verbose.
Two are more serious.
The assistant retrieves an outdated HR policy.
And one is critical.
A permission test fails.
An employee without executive access receives information from an executive compensation document.
The average score is still 92%.
The system is still not releasable.
This is why AI evaluation cannot be reduced to one percentage.
A system can have a high average score and still contain an unacceptable failure mode.
Pass Rate and Release Readiness Are Not the Same Thing
CedarWorks therefore adds severity.
| Failure | Severity | Release impact |
|---|---|---|
| Response slightly too verbose | Low | Acceptable |
| Missing citation | Medium | Investigate |
| Uses outdated policy | High | Block release |
| Invents nonexistent benefit | High | Block release |
| Exposes restricted document | Critical | Block release |
Now the release condition is no longer:
Overall score > 90%
It becomes something closer to:
High overall quality and zero critical failures.
That is a far more realistic way to think about trustworthy AI.
NIST’s evaluation work emphasizes that context matters when measuring AI systems; accuracy is only one trustworthiness characteristic alongside areas such as reliability, robustness, safety, security, privacy and harmful bias.
LLM Evaluation: Start With the Answer
Suppose Ask Cedar did not use RAG.
Imagine it simply took a user prompt and generated a response.
The evaluation problem would be relatively contained.
We might measure whether the response is:
correct,
relevant,
safe,
clear,
consistent with instructions,
and appropriately uncertain.
Some properties can be checked automatically.
If the required output is JSON, code can verify whether the JSON is valid.
If a classification must be one of five labels, exact matching may be appropriate.
Other properties are subjective.
For example:
“Was the explanation easy for a nontechnical employee to understand?”
That may require human reviewers or a carefully validated model-based grader.
The important principle is:
Choose the grader based on the thing being measured.
Do not use an LLM judge because “AI evaluates AI” sounds sophisticated.
If a deterministic check can answer the question reliably, use the deterministic check.
Human Evaluation Still Matters
Automated evaluation is attractive because it scales.
But humans remain important.
CedarWorks asks HR specialists to review a subset of Ask Cedar’s responses.
They score:
policy correctness,
completeness,
risk,
and whether the answer would actually help an employee.
Those expert examples become reference points for the automated evaluation system.
This is especially useful where “correct” is not a simple string match.
OpenAI’s current evaluation guidance similarly recommends combining expert-labelled examples with automated graders, particularly where the evaluation involves subjective or domain-specific criteria.
The goal is not:
humans or automated evaluation.
It is:
humans establish quality → automation helps measure it repeatedly.
RAG Changes the Evaluation Problem
Ask Cedar is not just an LLM.
It has retrieval.
That creates an important debugging problem.
Suppose the system gives the wrong answer.
What failed?
There are at least three possibilities.
Retrieval Failure
The correct document never reached the model.
The knowledge base contains the right policy, but search retrieved something else.
Context Failure
The correct document was retrieved, but irrelevant passages crowded the useful information out or the important passage was poorly extracted.
Generation Failure
The correct supporting information reached the model, but the model still produced an incorrect or unsupported answer.
Those failures can look identical to the user:
Wrong answer.
But they require completely different fixes.
Google Cloud’s current RAG evaluation guidance explicitly separates retrieval quality from generation quality for this reason. A RAG system may fail because search did not retrieve the correct material or because the model did not use the retrieved context correctly.
This is one of the most useful ideas in RAG evaluation.
Do not evaluate only the final sentence.
Evaluate the pipeline.
A Wrong Answer Can Have a Good Model Behind It
One Ask Cedar test asks:
“Can employees carry unused annual leave into the next calendar year?”
The correct policy exists.
But the assistant answers incorrectly.
The team initially considers changing the LLM.
Then they inspect the retrieval trace.
The model was given three chunks.
Chunk 1: an old 2024 leave policy.
Chunk 2: an unrelated payroll document.
Chunk 3: a holiday-calendar page.
The current 2026 leave policy was never retrieved.
The model is not the main problem.
Search is.
Changing to a more expensive language model might produce a more polished version of the same wrong answer.
That is why evaluation saves money as well as improving quality.
It prevents teams from treating every AI failure as a model problem.
The Three Questions I Would Ask Every RAG System
Instead of one “RAG accuracy” score, separate the problem.
| Layer | Question |
|---|---|
| Retrieval | Did we find the information needed to answer the question? |
| Grounding | Is the generated answer supported by the retrieved information? |
| Usefulness | Does the answer actually solve the user’s problem? |
These are related, but they are not interchangeable.
An answer can be grounded but useless.
For example, the retrieved policy may say:
“International remote work is subject to Policy HR-47.”
If the assistant responds:
“International remote work is subject to Policy HR-47.”
that response may be perfectly grounded.
But the employee still does not know whether they can work from Spain for three weeks.
Groundedness is necessary.
It is not the same as usefulness.
Citations Need Their Own Evaluation
A common mistake is to treat the presence of a citation as proof of reliability.
Ask Cedar produces:
Employees may carry five unused leave days into the following year.
Source: Annual Leave Policy.
Looks trustworthy.
But when the evaluator opens the cited document, it says nothing about carrying five days forward.
The citation exists.
It is just the wrong evidence.
So CedarWorks adds another check:
Does the cited source actually support the claim?
That is stronger than:
Did the response contain a citation?
The distinction matters because confident formatting can make unsupported AI output look unusually credible.
What Happens When the Answer Does Not Exist?
One of the most important evaluation cases is also one of the simplest:
Ask something that the system cannot answer.
For example:
“Does CedarWorks reimburse employees for personal streaming subscriptions?”
There is no such policy.
The desired answer is not a creative guess.
It is something like:
“I couldn’t find an approved CedarWorks policy covering personal streaming subscriptions.”
A system that knows when it lacks evidence is often more useful than a system that answers every question.
So “refusal to invent unsupported information” becomes an explicit evaluation criterion.
This changes the team’s mindset.
They are no longer trying to maximize the number of questions answered.
They are trying to maximize the number of questions handled correctly.
Those are not the same thing.
Then CedarWorks Adds an Agent
Several months later, Ask Cedar evolves.
Employees can now say:
“Book two days of annual leave next Thursday and Friday.”
The assistant can:
check the employee’s leave balance,
look at the calendar,
submit a leave request,
and report the result.
Ask Cedar is no longer only generating answers.
It is taking actions.
Evaluation now becomes substantially harder.
Agent Evaluation Has Two Stories
Suppose the agent successfully submits the leave request.
The final response says:
“Your leave request has been submitted.”
Success?
Maybe.
Look at what happened internally.
The agent:
checked the calendar,
queried the leave system,
called the leave API,
called the leave API again,
retried unnecessarily,
and eventually submitted the request twice before one duplicate was rejected.
The final answer looks correct.
The trajectory is poor.
For agentic systems, I would evaluate two stories:
| Evaluation | Question |
|---|---|
| Outcome evaluation | Did the agent achieve the user’s goal? |
| Trajectory evaluation | Did it take a safe and sensible path to achieve it? |
Google Cloud’s agent-evaluation tooling makes the same distinction between final-response evaluation and trajectory evaluation—the sequence of tools and actions used to reach the result.
This becomes increasingly important as AI systems gain authority.
A chatbot can say something wrong.
An agent can do something wrong.
The Correct Tool Matters
Imagine an employee asks:
“How many leave days do I have?”
The agent has two tools.
Tool A: read_leave_balance
Tool B: submit_leave_request
The correct path is obvious.
Read the balance.
Do not submit anything.
The evaluation should therefore verify not just the text response but whether the correct tool was selected.
Now imagine:
“Cancel my leave request for next Monday.”
Again, tool selection matters.
This is one reason an AI agent can produce a plausible final message while the workflow underneath is unsafe or inefficient.
Evaluate the System, Not Just the Model
By this point, CedarWorks has learned something important.
Ask Cedar’s production quality depends on far more than the language model.
It depends on:
the documents,
retrieval,
metadata,
permissions,
prompts,
tool definitions,
APIs,
business rules,
model behavior,
and monitoring.
That means:
Model evaluation is not the same thing as system evaluation.
A state-of-the-art model can sit inside a badly engineered system.
A smaller model can sit inside an excellent one.
The user experiences the system.
Not the benchmark score.
Digital Stackroom AI Release Gate
Before releasing an important AI system, I would evaluate it across six dimensions.
| Dimension | What we ask |
|---|---|
| Outcome | Did the system accomplish the user’s actual task? |
| Evidence | Are important claims grounded in appropriate information? |
| Behavior | Did it follow instructions, rules and expected tool usage? |
| Resilience | Does it handle ambiguity, missing data and unusual inputs safely? |
| Economics | Are latency and cost acceptable at real production volume? |
| Operations | Can failures be monitored, diagnosed and improved after release? |
This is deliberately broader than “accuracy.”
A system can produce accurate answers but be too expensive.
It can be fast but unsafe.
It can answer normal questions correctly while collapsing on edge cases.
It can succeed during launch week and silently degrade after the underlying knowledge base changes.
Release readiness is multidimensional.
Cost Belongs in Evaluation
Suppose Model A passes 96% of CedarWorks’ evaluation set.
Model B passes 95%.
Model A costs four times as much to operate.
Which model should the company choose?
There is no automatic answer.
If the extra 1% represents critical failures, Model A may easily be worth the cost.
If both models pass every high-severity case and the difference comes from writing style, Model B may be the better production choice.
This is why model selection should consider:
cost per successful task
rather than only:
cost per token
or:
benchmark score.
Evaluation turns that decision into something measurable.
Latency Belongs There Too
Ask Cedar produces a perfect answer.
It takes 38 seconds.
Employees stop using it.
Technically correct.
Operationally unsuccessful.
For interactive systems, quality must be evaluated alongside response time.
For background workflows, users may tolerate minutes.
For a voice assistant, even a few seconds of unnecessary delay may make the interaction feel broken.
There is no universal latency threshold.
There is a threshold appropriate to the job.
Again, context matters.
Don’t Build an Evaluation Dataset Made Entirely of Easy Questions
A surprisingly common mistake is building an eval that the system is designed to pass.
Imagine 100 questions that all follow the same pattern:
“What is the policy for X?”
That may test something.
But users will not cooperate with the test format.
They will write:
“Can I take my laptop overseas?”
“I’m moving to Dubai for six weeks, is that okay?”
“remote abroad?”
“My manager said it’s allowed but HR portal says something else.”
Real usage is messy.
Your evaluation data should be messy too.
Otherwise the evaluation measures performance in a laboratory that production users never visit.
Production Failures Should Feed the Test Set
After launch, an employee asks Ask Cedar:
“Can I use my learning allowance for a certification exam retake?”
The assistant answers incorrectly.
The problem is fixed.
But the work should not end there.
That question now enters the permanent regression suite.
The next prompt change, retrieval update or model upgrade must still answer it correctly.
This turns failures into assets.
Over time, the evaluation dataset becomes a record of what the organization has learned about its AI system.
OpenAI’s dataset guidance similarly recommends expanding evaluation data with edge cases and blind spots discovered over time, rather than treating the original test set as fixed.
Model Upgrades Should Trigger Evals, Not Excitement
A new model is released.
The benchmark numbers are better.
The marketing page says it reasons more effectively.
Should CedarWorks upgrade Ask Cedar?
Maybe.
But the safe workflow is not:
New model → deploy
It is:
New model → run the same evaluation suite → compare failures → evaluate cost and latency → decide
The newer model might improve difficult reasoning cases.
It might also behave differently around formatting, refusals, tool calls or verbosity.
Evals give the team a common language for comparing those changes.
OpenAI’s evaluation guidance specifically describes evals as a way to test whether applications continue meeting expectations when models, prompts or system configurations change.
One Score Can Be Useful—Just Don’t Worship It
Dashboards need summaries.
Leadership wants a trend.
So CedarWorks eventually produces a release score.
But the score sits on top of the detailed results.
It does not replace them.
The team might report:
Overall pass rate: 95%
Critical failures: 0
High-severity failures: 1
Grounding: 97%
Retrieval success: 94%
Permission tests: 100%
Median response time: 2.8 seconds
Average request cost: $0.03
Now “95%” has context.
Without the other numbers, it does not.
What About LLM-as-a-Judge?
One language model can evaluate the output of another.
This is useful.
It is also easy to misuse.
For CedarWorks, a model-based evaluator might score whether an answer is:
clear,
relevant,
complete,
or appropriately professional.
But before trusting that grader, the team compares its scores against HR experts.
If the automated grader consistently considers poor answers “good,” scaling it simply produces wrong judgments faster.
A grader is itself a system that needs evaluation.
That recursive idea sounds inconvenient.
It is also true.
Red Teaming Is Part of Evaluation, Not a Separate Universe
Normal evaluation asks:
“Does the system work under expected conditions?”
Red teaming asks something closer to:
“What happens when conditions become hostile, manipulative or unexpected?”
CedarWorks tests questions designed to:
override instructions,
request restricted information,
confuse the retrieval layer,
induce unsupported answers,
or manipulate tools.
The goal is not to replace normal evaluation.
It complements it.
NIST’s 2026 ARIA Evaluation Planning Manual describes holistic AI evaluation using a combination of model testing, red teaming and user testing, rather than relying on only one testing method.
That is a useful model for production teams.
User Testing Finds Problems Your Metrics Miss
CedarWorks eventually gives Ask Cedar to a pilot group of employees.
Something interesting happens.
The system’s evaluation scores are strong.
But employees repeatedly ask:
“Which policy did this answer come from?”
The assistant already displays citations.
The citations are just visually difficult to find.
That problem would never appear in a model-quality benchmark.
It appears immediately in user testing.
This is why real AI quality has a technical side and a human side.
An accurate system that users cannot understand or trust may still fail.
When Is the System Ready?
After several evaluation cycles, Ask Cedar reaches:
zero critical permission failures,
zero known unsupported policy claims in the release suite,
high retrieval success,
strong citation support,
acceptable latency,
acceptable cost,
and satisfactory pilot-user feedback.
Is it now perfect?
No.
That is not the standard.
The standard is:
Do we have enough evidence that the system performs acceptably for its intended use, and do we have mechanisms to detect when it does not?
That is a much more realistic definition of production readiness.
The Mistake Is Asking “Which Model Is Best?”
Teams often begin their AI journey with:
“Which model should we use?”
A better sequence is:
“What does success look like?”
Then:
“How will we measure it?”
Then:
“Which architecture and model achieve that outcome reliably enough?”
Evaluation turns model selection from an opinion contest into an engineering decision.
The Framework I Would Keep
If you remember only one thing from this article, use this:
Test Set → Run → Measure → Diagnose → Improve → Regression Test → Release → Monitor
And underneath that process, evaluate six things:
Outcome → Evidence → Behavior → Resilience → Economics → Operations
The first line tells you how to improve the system.
The second tells you what not to forget.
Together, they turn evaluation from a final QA step into part of the architecture.
Final Thoughts
The most dangerous AI system is not necessarily the one that obviously fails.
Obvious failures are easy to notice.
The harder problem is the system that works beautifully 95% of the time and fails confidently during the 5% that matters most.
That is why a good demo proves very little.
A demo tells you:
“This system can work.”
Evaluation asks:
“Under what conditions does it work, when does it fail, how serious are those failures, and will we notice when they happen?”
Those questions are less exciting than a live demo.
They are also what turns AI from a prototype into a product.
CedarWorks didn’t make Ask Cedar reliable by finding one magical metric.
It did it by repeatedly converting uncertainty into evidence:
test the system,
measure it,
understand the failures,
fix the right component,
test again,
and keep doing that after release.
That is AI evaluation.
And as AI systems move from answering questions to retrieving private information, calling tools and taking real actions, it may become one of the most important engineering disciplines around them.
Official Sources & Further Reading
NIST — AI Measurement and Evaluation
NIST’s work on metrics, testing and evaluation for trustworthy AI systems.
NIST — AI Risk Management Framework
Framework covering governance, mapping, measurement and management of AI risks throughout the lifecycle.
NIST — ARIA Evaluation Planning Manual
A 2026 framework describing holistic evaluation across model testing, red teaming and user testing.
Google Cloud — Master Generative AI Evaluation: From Single Prompts to Complex Agents
Practical guidance covering LLM, RAG and agent evaluation.
Google Cloud — RAG Systems: Best Practices for Evaluation
Guidance on diagnosing retrieval quality separately from generated-answer quality.
Google Cloud — Agent Evaluation
Documentation describing final-response and trajectory-based evaluation for agentic systems.
OpenAI — Evaluation Best Practices
Guidance on defining evaluation objectives, building representative datasets, creating graders and continuously testing AI applications.
OpenAI — Datasets and Evaluations
Guidance on building evaluation datasets, annotations, graders and regression-testing workflows.