Testing a customer service chatbot before launch should combine automated evaluation against a labeled test set with structured human review, since neither approach alone catches every failure mode. Start by assembling a test set of at least a few hundred real or realistic customer questions, spanning common issues, edge cases and adversarial attempts to get the bot to say something wrong or inappropriate, each paired with a verified correct answer or expected behavior. Automated scoring, including model-as-judge grading for open-ended responses, can run this test set quickly and repeatedly, but should be spot-checked against human judgment since judge models carry their own blind spots. Beyond answer accuracy, test the escalation logic specifically, confirming the bot hands off correctly on low confidence, explicit requests for a human, and out-of-scope topics, since a broken handoff is often worse than a wrong answer. A staged rollout, starting with internal employees, then a small percentage of real traffic with close monitoring, then full traffic, catches issues a pre-launch test set misses under real usage patterns. Re-running the full evaluation suite after any prompt, model or knowledge base change keeps quality from silently drifting. Nanobase AI builds this evaluation suite as a permanent asset the client keeps using after launch, not a one-time pre-launch checklist.
Start with the rubric, not the test questions
Most pre-launch chatbot testing efforts jump straight to writing test questions, which produces a pile of pass or fail cases without a clear definition of what "pass" actually means beyond a subjective read. A proper evaluation starts by defining the rubric first: the specific dimensions a good answer needs to satisfy, scored independently, so a response that gets the facts right but skips a required escalation trigger is caught as a distinct failure rather than blended into one vague accuracy score. Separating factual correctness, tone and policy adherence, and escalation behavior into independent scoring dimensions reveals exactly which part of the system is weak, instead of a single pass rate that hides whether the bot is wrong on facts or wrong on judgment.
The four dimensions worth scoring separately
| Dimension | What it checks | How it's typically scored |
|---|---|---|
| Factual accuracy | Does the answer match the verified source content | Automated comparison against a labeled answer, spot-checked by a human |
| Escalation correctness | Does the bot hand off when it should, and only when it should | Rule-based check on trigger conditions plus manual review of edge cases |
| Tone and policy adherence | Does the response stay within brand voice and avoid unauthorized commitments | Model-as-judge scoring, calibrated against human review |
| Refusal correctness | Does the bot decline out-of-scope requests appropriately without over-refusing | Manual review, since over-refusal is easy for automated judges to miss |
Model-as-judge grading speeds up scoring on open-ended dimensions like tone, but it should be calibrated against a human-reviewed sample before being trusted at scale, since judge models carry their own blind spots that mirror the model being evaluated.
Adversarial cases the standard test set misses
A test set built only from realistic customer questions catches accuracy problems but misses the failure modes that actually cause damage after launch. Deliberately adversarial cases belong in every pre-launch suite: attempts to get the bot to issue an unauthorized discount, prompts that try to override its instructions through claimed authority or fake system messages, questions that push it toward legal or medical territory outside its scope, and simulated hostile or upset customers to check tone under pressure. Skipping adversarial testing because the friendly test cases all passed is the single most common reason a chatbot performs well in a demo and then fails publicly within its first week of real traffic, since real customers, unlike a curated test set, actively look for the system's edges.
Wiring evaluation into a regression pipeline
The test suite loses most of its value if it only runs once before launch, since prompts, models and knowledge base content all continue changing after go-live. Treating the evaluation suite as a regression pipeline, similar to how software teams treat automated tests, keeps quality visible over time:
- Run the full rubric-scored suite before any prompt, model or knowledge base change ships.
- Block the change if scores on any dimension drop below the prior baseline by more than a defined margin.
- Add every real production failure discovered post-launch back into the suite as a new test case.
- Re-baseline periodically as the bot's scope intentionally expands.
Frequently asked questions
How large should a pre-launch test set be?
A useful starting point is a few hundred cases spanning common questions, edge cases and adversarial attempts, though the right size depends more on how varied your support volume is than on hitting a specific number.
Can model-as-judge scoring fully replace human review?
No, model-as-judge scoring works well for speed and repeatability on well-defined dimensions, but it should be periodically checked against human judgment since judge models share blind spots with the models they evaluate.
What's the biggest gap in most pre-launch testing efforts?
Adversarial and edge-case testing is the most commonly skipped step, since teams naturally gravitate toward testing the questions they expect customers to ask rather than the ones designed to break the system.
Should the evaluation suite run automatically in a CI pipeline?
Yes, wiring the suite into the same deployment pipeline used for prompt and code changes catches regressions before they reach customers, the same discipline software teams already apply to application code.
How Nanobase AI helps
Nanobase AI, a Silicon Valley enterprise AI engineering company, builds the rubric, the adversarial test cases and the regression pipeline as one connected system, then hands it to the client as a permanent asset rather than a one-time pre-launch checklist. Teams can see this evaluation approach applied live in a product demo before committing to a build.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.