A good target for a chat assistant is roughly thirty to sixty output tokens per second per user stream, since average adult reading speed is around two hundred to two hundred fifty words per minute, or about four to five tokens per second, and generation noticeably faster than that reading pace is what makes streaming output feel instantaneous rather than merely acceptable. Below about fifteen to twenty tokens per second, users tend to perceive the assistant as sluggish even with streaming, because the visible text lags behind natural reading pace. Time to first token matters as much as steady-state speed for perceived responsiveness: a sub-second TTFT followed by thirty tokens per second generally feels faster than a fast steady rate preceded by a multi-second delay. Coding assistants, voice agents, and agentic tool-calling loops often need higher rates, sometimes eighty to one hundred fifty tokens per second, because latency compounds across multiple sequential model calls. The right number ultimately depends on the use case and the model's reasoning overhead, since chain-of-thought and reasoning models generate many more tokens before a useful answer appears. Nanobase AI, a Silicon Valley applied AI company, sets latency targets per use case rather than applying one blanket number across a product.
Why one universal number misleads
A single "good tokens-per-second" number gets quoted constantly, but it collapses several genuinely different requirements into one figure. A voice assistant, a coding tool, and a document summarizer all have different tolerance for generation speed because the user's interaction pattern with the output is different in each case, not because one model is objectively faster than another.
Set your target from the use case's interaction pattern first, then work backward to what model size, quantization, and hardware combination actually hits it, rather than starting from a generic target and hoping it fits.
Targets mapped to use case
| Use case | What drives the target | Typical target range |
|---|---|---|
| General chat assistant | Should meaningfully exceed average adult reading speed for streaming to feel instant | Comfortably above reading pace, with sub-second time to first token |
| Coding assistant (autocomplete/chat) | Latency compounds across multiple sequential suggestions and edits | Needs to feel faster than chat, since developers reject slow completions quickly |
| Voice agent | Total round-trip latency, including speech-to-text and text-to-speech, must stay conversational | Requires very low time to first token above all, since any generation delay is audible as dead air |
| Agentic tool-calling loop | Multiple sequential model calls compound; each one's latency multiplies | Needs high per-call speed since the user waits through several calls, not one |
| Reasoning models (chain-of-thought) | Generates substantially more tokens before a final answer appears | Needs a higher raw tokens-per-second just to match the perceived latency of a non-reasoning model |
| Batch or offline summarization | No user waiting in real time | Throughput per GPU matters more than per-request speed |
Why reasoning models break the naive comparison
A reasoning model that produces a chain of thought before its final answer needs a substantially higher tokens-per-second rate than a non-reasoning model just to deliver an equivalent perceived response time, because the user is effectively waiting through all of the reasoning tokens plus the answer, not just the answer. This is the single most common reason a deployment that "used to feel fast enough" starts getting latency complaints after a model upgrade to a newer reasoning-capable version: the model's answer quality may have improved, but its token generation volume per response also grew, and the serving configuration was not adjusted to compensate.
Levers that move the achievable rate, ranked by typical impact
- Model size: A smaller model at the same precision generates faster per GPU, all else equal, since there is less compute per token.
- Quantization: Moving from FP16 to FP8 or INT4 reduces compute and memory bandwidth per token, typically improving generation speed meaningfully, with a tradeoff to validate against output quality.
- Batch size and concurrency: Per-request tokens-per-second typically decreases as concurrent load increases, since compute is shared across more simultaneous sequences; this is a deliberate throughput-versus-per-request-latency tradeoff, not a bug.
- Hardware generation: Moving from H100 to H200 or B200 increases available compute and memory bandwidth, which raises the achievable ceiling for both throughput and per-request speed.
- Speculative decoding: Can meaningfully raise effective per-request speed at lower concurrency without changing the underlying model, as covered in speculative decoding.
Setting and validating your own target
- Identify the interaction pattern: is a human reading the output in real time, listening to it, or waiting through a multi-step pipeline before seeing anything.
- For human-read text, benchmark against natural reading speed as the floor, not the target, since generation should exceed reading pace for streaming to feel instant rather than merely tolerable.
- For voice and highly interactive use cases, prioritize time to first token over steady-state rate, since delay before any response starts is what users notice most.
- For multi-step agentic pipelines, model total latency across all sequential calls, not any single call's speed in isolation.
- Validate the target under realistic concurrency, not a single-request benchmark, since real deployments rarely run at concurrency one.
Frequently asked questions
Does a higher tokens-per-second number always mean a better user experience?
Not necessarily. Time to first token, consistency of the token stream, and how well the interaction pattern matches the use case all matter alongside raw speed; a very fast rate with a long initial delay can feel worse than a moderate rate that starts immediately.
How much does output length itself affect perceived speed?
Significantly. A short, direct answer at a moderate rate can feel faster overall than a long, verbose answer at a high rate, since perceived responsiveness includes total time to a useful answer, not generation speed in isolation.
Should agentic applications target a different speed than chat applications?
Yes, generally higher, since agentic loops chain multiple model calls together and any per-call latency compounds across the whole task; a speed that feels fine for one chat response can feel slow when repeated several times in sequence.
Is it worth measuring tokens-per-second separately from time to first token?
Yes, always. They capture different parts of the user experience and respond to different tuning levers, so tracking them as one blended metric obscures which lever actually needs attention when performance issues arise.
How Nanobase AI helps
Nanobase AI sets latency and throughput targets per use case rather than applying one blanket number across a product, then selects model size, quantization, and hardware to hit that target under realistic concurrency. This use-case-specific approach is part of Nanobase AI's broader inference engineering practice.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.