How to build evals for AI agents from your real failures

Most teams test an AI agent by chatting with it a few times before launch. That works for a chatbot, where the reply is the whole product. It fails for an agent, because an agent books appointments, updates CRM records, and issues refunds across many steps, and the reply tells you nothing about whether those actions happened. So the failures surface in production, one complaint at a time, and every prompt change or model upgrade is a guess.

Take a home builder with an agent that books model home tours. A visitor asks for Saturday at 10 and the agent replies "You are booked for Saturday at 10, see you there." The logs show it checked the open slots and then wrote the confirmation without ever calling the calendar tool. The reply was perfect and the booking did not exist.

Evals close this gap. An eval is an automated test that runs your agent on a fixed set of tasks and checks what it did. Here is how to build a set you can trust, using failures you already have.

Grade the end state

A chatbot answers one question and you compare the answer to a reference. An agent works differently. It reasons, calls a tool, reads the result, changes something in a CRM or a calendar, and repeats this for ten or twenty steps. Two runs of the same request can take different routes and both be correct, and one wrong step early can spoil everything after it.

So the test for the booking agent ignores the confirmation message and opens the calendar. If an event exists for Saturday at 10 with the right community and the visitor's phone number, the run passes. If the calendar is empty, the run fails, however polite the reply was. Anthropic's engineering team makes the same point in its guide to agent evals with a flight-booking agent, where the result that counts is the reservation row in the database.

Resist the urge to also check the exact order of tool calls. Anthropic's team found that approach too rigid, because agents regularly find valid routes that the test author did not expect. If the tour is on the calendar and the lead is in the CRM once, you do not care which question the agent asked first.

Build the first set from failures you already have

You do not need hundreds of tasks. Anthropic's guide says 20 to 50 simple tasks drawn from real failures is a good start, because early changes to an agent have large effects and large effects show up in small samples. Your raw material is already sitting in the support inbox, the thumbs-down log, the sales team's complaints, and the chat transcripts where a visitor typed "that is not what I asked."

Turn each failure into a task in four steps.

Freeze the input. Save the visitor's message and a snapshot of the calendar and CRM as they were at that moment.
Write down the outcome that should exist afterwards, such as one calendar event, one CRM lead, and no duplicates.
Group similar failures. Forty phantom bookings are one bug, and one bug needs one or two tasks, so you write one task per distinct condition.
Run the new task against the version of the agent that had the bug. It must fail there. A test that passes on the broken version tests nothing.

Then add the opposite cases. If every task rewards booking, you end up with an agent that books everything, including a tour at a community that sold out last month. Anthropic's team hit this with web search in Claude.ai, where they needed tasks for "should search" and tasks for "should answer from memory" before the behavior settled. For the booking agent, pair each should-book task with a should-not-book task (a sold-out community, a date in the past, a visitor who only wanted the price sheet).

One more test before a task goes in. Two people who know the business should read it and agree on pass or fail without discussion. If they argue, the task is ambiguous, and an ambiguous task produces noise that you will later mistake for a model problem.

Hold everything steady except the agent

An eval compares runs, and a comparison only works when the agent is the single thing that changed. Each run should start from a clean copy of the test CRM and calendar. Anthropic's team reports a case from its internal evals where Claude scored better on some tasks because it could read the git history left behind by earlier trials, which is the eval version of leaving the answer key on the desk.

Anmol Tukrel's "Evals for AI Agents" talk at Anthropic (November 2026) sorts the fixes into three groups, and they map well to marketing systems.

Things you control, you freeze. Use a test CRM account, a database snapshot, and a pinned clock, because "this Saturday" means a different date every week and your expected answer should not expire.
Things you do not control, you replay. If the agent calls a mortgage rate API, call it once, save the response, and serve that saved response on every run.
When neither works, you loosen the grader. If the agent estimates a monthly payment from live data, accept any figure within 2% of the value the tool returned.

Run the real agent in the eval, with the production prompt, tools, and settings. A simplified copy is easier to wire up, and it measures something you do not ship.

Use the cheapest grader that works

Use a code grader for anything with a checkable answer, such as "a calendar event exists" or "the lead went to the Carlsbad sales rep." It is rigid, so confirm it does not fail a correct answer over formatting.
Use a model judge for open-ended replies, such as an answer to a financing question. It has its own error rate, so check it against people.
Use a human to build the task set and spot-check the judge. A human is slow and costly, so keep people out of the per-run gate.

Start with code. Code graders are fast, free to rerun, and give the same verdict every time. Write separate checks for separate steps, so a failure tells you where the agent went wrong. For a lead-routing task that means one check for the zip code it extracted, one for the territory rule it applied, and one for the rep it assigned. "Routing failed" sends you to the transcript, while "zip code right, territory rule wrong" sends you to the fix. Keep the checks tolerant as well, because Anthropic's article on writing tools for agents warns against verifiers that reject correct responses over punctuation or phrasing.

Use a model judge when there is no single right answer. Say a visitor asks whether the builder has financing for first-time buyers. Do not ask the judge for a score from 1 to 5, because nobody can say what a 3 means. Ask yes-or-no questions with one judge call per question, such as whether the reply mentioned the preferred lender program, whether it avoided quoting a specific interest rate, and whether it offered a next step.

Then check the judge. Tukrel's talk gives a workable routine. Hand-label 50 to 100 real outputs yourself, let the judge grade the same set, and compare. If the judge agrees with you less than about 85% of the time on a criterion, rewrite that criterion or split it in two. The talk is clear that 85% is a rule of thumb. Keep the judge on a fixed model, and never let the model under test grade its own work.

Read transcripts before you believe a number

The same agent gives different results on the same input, so one run per task tells you little. Run each task several times (five is a reasonable start) and look at how often it passes. The arithmetic is unkind. An agent that books correctly 80% of the time gets three bookings in a row right only 51% of the time (0.8 x 0.8 x 0.8). If your product promise is "it works every time," the second number is the one your customers experience.

Small suites are also noisy. With 30 tasks, a single task that flips from pass to fail moves the score by 3.3 points, so a prompt change that "improved quality by 4 points" may have improved nothing. Rerun before you celebrate.

And sometimes the eval itself is broken. Anthropic's guide describes Opus 4.5 scoring 42% on a benchmark called CORE-Bench until a researcher found graders that marked "96.12" wrong because they expected "96.124991". After the fixes and a less constrained scaffold, the score was 95%. The same guide says a task that fails on every one of many trials is usually a broken task. You find these problems by opening the transcripts and reading what happened, which is tedious and is also where most of the useful findings are.

Keep two suites

Sort your tasks into two piles. The capability suite holds hard tasks that the agent mostly fails today, such as a visitor who changes the tour date twice and then asks about a different community. Its pass rate should be low, because a suite at 100% cannot show you an improvement. The regression suite holds tasks the agent already passes reliably, and it should stay close to 100%. A drop there means you broke something.

Tasks move between the piles. When the agent passes a hard task consistently, promote it to the regression suite and write a harder one to take its place. The phantom booking test starts as a capability task and, once fixed, becomes a permanent guard against the bug coming back.

What you get on model release day

New models arrive every few months, and each one can change how your prompts behave. Without evals you have two bad options, which are to spot-check five chats and ship on a feeling, or to wait three weeks and test by hand. With evals the routine takes an afternoon. Point the harness at the new model with the same prompts, tools, and graders. Run the regression suite to see what broke, run the capability suite to see what became possible, read the transcript of every regression, and then decide to ship, hold, or tune.

Track cost and latency per task next to quality, because a model switch often changes the bill more than the score. Anthropic published one example from an internal customer support benchmark of 44 tickets. The starting setup scored 74.4% at 4.6 cents per ticket. After a prompt cleanup and a move to a smaller, newer model, the final setup scored 98.9% at about 1 cent.

Those two scores come from the 30 tickets used for tuning, so the honest number is the one from the 14 tickets held back, where accuracy went from 78.6% to 90.5%. Do the same with your own set. If you tune prompts against the tasks you score on, the score rises while production stays flat. Keep a portion of tasks that nobody (and no tuning script) reads, and trust only the improvement that shows up there.

Start with 20 failures this week

On day one, pull 20 real failures from tickets and chat logs and write one row for each with the input, the expected outcome, and the reason it is hard. On days two and three, wire them to your real agent, with code checks where the answer is checkable and judge questions where it is not. On days four and five, run five trials per task on your current model and on the newest one, and read every failure.

If you build on Claude, Claude Code has two commands that do much of this work. According to Anthropic's developer blog, /claude-api build-eval interviews you, builds the eval inside your codebase, and waits for your approval of the cases and the grader. /claude-api hillclimb then improves your agent against that eval one change at a time and uses held-out cases to catch overfitting.

SmartMetrics builds AI agents for lead generation and CRM workflows. If you want a second pair of eyes on your first eval set, send us your 20 failures.