The core KPIs for an AI support agent fall into four groups, resolution quality, efficiency, cost and customer experience, and tracking all four together prevents optimizing one at the expense of another. Resolution quality includes containment rate, the share of conversations fully resolved without human escalation, and accuracy on a sampled or human-reviewed set of answers, since a high containment rate built on wrong answers is a false win. Efficiency metrics include average handle time, first-response time, and deflection rate for tickets that never needed a human at all. Cost metrics should track cost per resolved conversation and total API or infrastructure spend against ticket volume, so cost trends are visible before they become a problem. Customer experience metrics, CSAT specifically on AI-handled conversations, escalation rate by reason category, and repeat-contact rate for the same issue, reveal whether customers are actually satisfied or just not complaining. Reviewing these KPIs weekly during the first months after launch, then monthly once the system stabilizes, catches drift in either the knowledge base or model behavior early. Nanobase AI sets up this KPI dashboard as part of every deployment so performance is measurable from the first week live.
A list of KPIs is not a dashboard until it has a review cadence
Knowing that containment rate, accuracy, handle time and CSAT all matter is not the same as having a system that surfaces problems before they compound, because a KPI checked quarterly catches a problem three months after it started, while the same KPI checked weekly catches it in days. Structuring the dashboard around how often each metric needs review, not just which metrics exist, is what turns tracking into actual early warning.
KPI structure with review cadence
| KPI | How measured | Review cadence during rollout | Cadence once stable |
|---|---|---|---|
| Containment rate | Conversations fully resolved without escalation, by category | Weekly | Monthly |
| Answer accuracy | Sampled or human-reviewed correctness on live conversations | Weekly | Monthly |
| First-response time | Time from customer message to first AI response | Weekly | Monthly |
| Average handle time | Total conversation duration for AI-handled interactions | Weekly | Monthly |
| Cost per resolved conversation | Total AI system cost divided by resolved volume | Monthly | Monthly |
| CSAT on AI conversations | Post-interaction survey, scoped to AI-handled conversations | Weekly | Monthly |
| Escalation rate by reason | Share of conversations escalated, broken down by trigger category | Weekly | Monthly |
| Repeat-contact rate | Customers contacting again about the same issue within a set window | Weekly | Monthly |
Reviewing all eight metrics weekly during the first months after launch, then relaxing to monthly once the system stabilizes, catches drift in either the knowledge base or model behavior while it is still cheap to fix.
Why these group into four categories, and why that grouping matters
Resolution quality metrics, containment rate and accuracy, efficiency metrics, handle time and first-response time, cost metrics, and customer experience metrics, CSAT and escalation and repeat-contact rate, need to be reviewed together rather than in isolation, because optimizing one group at the expense of another produces a system that looks good on paper and performs poorly for customers. A high containment rate built on wrong answers is a resolution quality failure disguised as an efficiency win, and a low cost per conversation achieved by discouraging escalation is a cost win built on a customer experience failure. Reviewing all four categories side by side in the same dashboard session prevents any single metric from being optimized in isolation.
Setting up the dashboard practically
- Instrument the conversation system to log category, resolution status, and trigger reason for every interaction from day one, since retrofitting this logging after launch loses historical data.
- Connect a lightweight reporting layer, even a shared spreadsheet initially, that pulls these logs into the eight-metric structure above.
- Assign clear ownership: who reviews the dashboard weekly, and what threshold triggers a deeper investigation into a specific metric.
- Segment every metric by conversation category from the start, not just as an overall average, since category-level detail is where real problems are visible.
- Revisit the review cadence itself after the first quarter, moving stable metrics to monthly review while keeping any metric still showing volatility on a weekly cycle.
Logging category and resolution status from day one is the single step most teams regret skipping, since there is no way to backfill that detail once the conversation history has already happened without it.
Frequently asked questions
Which single KPI matters most if we can only track one?
Cost per resolved conversation, combined with accuracy, gives the closest single signal to overall system health, since it captures whether the AI is genuinely resolving issues cost-effectively rather than just appearing busy or fast.
How do we measure accuracy without reviewing every single conversation?
Sample a percentage of live conversations for human review, weighted toward higher-risk categories, and supplement with an automated LLM-as-judge grading pass against your golden test set for continuous coverage between manual reviews.
Should KPI targets be the same across all conversation categories?
No, a technical troubleshooting category and a simple order-status category will naturally have different containment rate and handle time expectations, so set category-specific targets rather than one blanket target across the whole system.
What signals that a KPI needs weekly review again after being relaxed to monthly?
Any sudden shift, a containment rate drop, a CSAT dip, or a spike in a specific escalation reason, should immediately return that metric to weekly review until the cause is identified and resolved.
How Nanobase AI helps
Nanobase AI sets up this KPI dashboard as part of every deployment, with the logging and category segmentation built in from day one so performance is measurable from the first week live rather than retrofitted later. This dashboard work connects directly to calculating ROI and to understanding CSAT impact using the same underlying data.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.