Evaluating an AI agent before production requires testing three separate layers: the underlying model's task accuracy, the agent's tool-calling correctness, and the end-to-end workflow's success rate on realistic scenarios. Start by building a labeled evaluation set of at least fifty to a few hundred representative tasks pulled from real user requests, then measure whether the agent selects the correct tools, passes correct arguments, and reaches the correct final outcome, not just whether it produces plausible-sounding text. Automated LLM-as-judge scoring can grade open-ended outputs at scale, but should be validated against human judgment on a sample before being trusted, since judge models have their own biases. Adversarial and edge-case testing matters as much as the happy path, including malformed inputs, ambiguous requests, and prompt injection attempts if the agent has web or document access. Before go-live, run the agent in shadow mode alongside human operators for a period, comparing its proposed actions to what a human actually did, and track cost and latency per task alongside accuracy. Nanobase AI builds this evaluation harness as a standing asset so agent quality can be regression-tested every time the prompt, model or tools change.
Build the pipeline, not a one-time test
Teams that treat evaluation as a pre-launch checklist tend to see quality regress silently the first time a prompt or model changes. The more durable approach is a standing pipeline with defined stages, each producing a metric tracked over time and re-run automatically whenever the agent changes. A pipeline with stages catches regressions before customers do; a one-time test only proves the system worked once, on the day you tested it.
| Stage | What it measures | When it runs |
|---|---|---|
| Offline eval | Tool selection accuracy, argument correctness, task success on a labeled set | Every prompt, tool or model change |
| LLM-as-judge scoring | Open-ended output quality, validated against human sample | Alongside offline eval, on a rolling sample |
| Shadow mode | Agent's proposed action vs what a human actually did, no execution | Weeks before go-live |
| Canary rollout | Real execution on a small traffic slice, with a human safety net | First weeks of production |
| Production monitoring | Cost, latency, success rate, safety violations at scale | Continuously post-launch |
Building the labeled set is the unglamorous part that matters most
None of the later stages matter without a labeled evaluation set of realistic tasks, the step teams most often shortcut. A useful starting set has fifty to a few hundred tasks pulled from actual user requests or historical cases, not synthetic examples, because real requests carry the ambiguity and edge cases synthetic sets miss. Each task needs a known correct outcome, or at minimum a clear rubric a human judge can apply consistently. The evaluation set should grow every time production surfaces a case the agent got wrong, turning every incident into a permanent regression check rather than a one-off fix.
Why LLM-as-judge needs a human check before you trust it
Using a model to grade another model's open-ended output scales evaluation far beyond what human review alone could cover, which makes it attractive for grading things like the tone or completeness of a drafted email. The risk is that judge models carry their own biases, such as favoring longer or more confidently worded answers regardless of correctness, and those biases go unnoticed until someone checks the judge against actual human ratings. The practical safeguard is to sample a subset of judged outputs, typically fifty to a hundred, have a human rate them independently, and measure agreement before trusting the judge's scores at scale. Re-running this calibration whenever the judge model or grading prompt changes keeps the automated scores meaningful rather than just plausible-looking numbers.
Shadow mode and canary rollout as the bridge to production
Neither stage substitutes for the other: shadow mode proves the agent's judgment without any execution risk, and canary rollout proves the system holds up once real actions are on the line. The sequence:
- Run the agent alongside real incoming tasks without letting it execute anything, logging what it would have done.
- Compare its proposed actions to what the human operator actually did, flagging every disagreement for review, not just the clear errors.
- Once disagreement rate and severity are acceptable, move to a canary rollout on a small slice of real traffic with execution enabled and a fast human override available.
- Expand traffic gradually, watching cost, latency and error rate at each step rather than jumping straight to full volume.
Testing the edges, not just the happy path
An agent that performs well on clean, well-formed requests can still fail badly on the inputs a human would never think to test, including malformed data, contradictory instructions, and, for any agent with document or web access, prompt injection attempts embedded in content it processes. Adversarial and edge-case scenarios belong in the same evaluation set as ordinary tasks, weighted according to how often they realistically occur, so a launch decision reflects performance across the full input distribution rather than only the cases a developer thought to write by hand.
Frequently asked questions
How big does the evaluation set need to be before launch?
There is no universal number, but fifty to a few hundred representative, real-derived tasks is a reasonable starting range. What matters more than the count is that the set reflects the actual distribution of requests the agent will see, including edge cases, not only clean examples.
What is the difference between shadow mode and a canary rollout?
Shadow mode runs the agent against real tasks without letting any action execute, purely to compare its decisions to what a human did. A canary rollout lets the agent actually execute on a small slice of real traffic, which is a stronger test but carries real risk, so it should only start after shadow mode results look acceptable.
How often should the evaluation suite be re-run?
On every change to the prompt, tool set or underlying model, at minimum, since any of these can shift behavior in ways that are not obvious from a quick manual check. Many teams also run it on a schedule to catch drift from upstream model updates they did not initiate themselves.
Can we skip shadow mode if we are confident in the agent?
It is possible but risky, since shadow mode is the cheapest stage at which to catch a systematic error before it touches a real customer or record. Skipping it shifts that discovery cost into production, where mistakes are harder and more expensive to undo.
How Nanobase AI helps
Nanobase AI builds this evaluation pipeline as a standing asset for every agent it deploys, so quality can be regression-tested every time a prompt, model or tool changes rather than re-verified from scratch. This work is closely tied to the observability and tracing infrastructure that supplies the data these evaluations run on, and to defining KPIs for agents already in production.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.