Making an AI voice agent sound natural with low latency requires optimizing every stage of the pipeline, speech-to-text, language model reasoning, and text-to-speech, for streaming rather than waiting for each stage to fully complete before starting the next. Practically, this means using a streaming speech-to-text model that returns partial transcripts as the customer speaks, a language model call that starts generating a response before the full transcript is finalized when possible, and a text-to-speech model that begins speaking the first words of a response while the rest is still being generated. Total round-trip latency under about 500 to 800 milliseconds is the threshold where conversation starts to feel natural rather than sluggish, and hitting that consistently usually requires colocating these services on shared infrastructure rather than chaining calls across separate third-party APIs with unpredictable network latency. Naturalness also depends on smaller details, allowing the caller to interrupt the agent mid-sentence, adding brief filler responses while a backend lookup runs, and choosing a text-to-speech voice with natural pacing rather than a flat, robotic cadence. Nanobase AI, an NVIDIA Inception Program member, tunes and often colocates this full pipeline on dedicated GPU infrastructure specifically to hit these latency targets reliably.
Total latency is a budget, not a single number to optimize blindly
Aiming for "low latency" without breaking the target down by pipeline stage leads teams to over-optimize one component, often the language model, while a slower speech-to-text or text-to-speech stage quietly eats the rest of the budget. Allocating the roughly 500 to 800 millisecond total latency target across each stage explicitly shows exactly where engineering effort will actually move the needle.
A representative latency budget
| Stage | Typical target | Optimization lever |
|---|---|---|
| Speech-to-text (partial transcript) | 100 to 200 ms | Streaming model returning partial results, not waiting for full utterance |
| Language model first token | 150 to 300 ms | Streaming generation, starting before the full transcript finalizes when safe to do so |
| Text-to-speech first audio | 100 to 200 ms | Streaming synthesis, speaking the first words while the rest generates |
| Network and orchestration overhead | 50 to 150 ms | Colocating services rather than chaining calls across separate third-party APIs |
Colocating the speech-to-text, language model and text-to-speech stages on shared infrastructure removes network round-trip overhead between separately hosted third-party APIs, which is often the single largest lever available once each individual model is already reasonably fast.
Naturalness is more than raw speed
Hitting the latency budget is necessary but not sufficient for a conversation that feels natural. Barge-in handling, letting the caller interrupt the agent mid-sentence and having the system stop immediately rather than finishing its planned response, matters as much as raw speed, since an agent that talks over an interrupting caller feels robotic regardless of how fast it responded. Brief filler responses, a short acknowledgment while a backend lookup runs longer than the latency budget allows, prevent dead air that feels broken even when the system is working correctly. Voice pacing and prosody, avoiding a flat, uniformly paced delivery, also affects perceived naturalness independent of latency.
A step-by-step approach to hitting the budget
- Measure current latency per stage individually before optimizing anything, using real telephony-quality audio rather than clean microphone recordings.
- Convert the slowest stage to streaming first, since a single non-streaming stage in an otherwise streaming pipeline caps the whole system's responsiveness.
- Test colocating services that are currently chained across separate providers, measuring the actual network overhead removed.
- Add barge-in handling and verify it works under real interruption timing, not just a scripted test.
- Re-measure total latency under concurrent call load, since a pipeline tested one call at a time often slows meaningfully once GPU and API capacity is shared.
Step two, converting the slowest non-streaming stage first, is usually where teams find the largest single latency improvement in the entire process.
Frequently asked questions
Is 500 to 800 milliseconds a hard requirement or a rough guideline?
It is a rough guideline based on where conversations start to feel natural rather than sluggish; going somewhat above it does not necessarily break the experience, but consistently exceeding it noticeably makes the agent feel slow compared to a human conversation.
Does self-hosting the models always improve latency compared to third-party APIs?
Often yes, since self-hosting removes network round-trips to external providers and allows colocating stages on shared hardware, though it requires the infrastructure and operational capability to run and tune the models yourself.
How much does concurrent call volume affect latency?
Significantly; a pipeline that hits the latency target for a single call can degrade under load if GPU or API capacity is shared across many simultaneous callers, which is why load testing at expected concurrency matters more than single-call benchmarks.
Should we optimize for lowest latency or highest transcription accuracy first?
It depends on the use case; a real-time conversational agent should prioritize latency with streaming models, while a use case tolerant of a short delay can prioritize accuracy instead, so this decision should follow from how the voice agent is actually used.
How Nanobase AI helps
Nanobase AI, an NVIDIA Inception Program member, tunes and often colocates this full pipeline on dedicated GPU infrastructure specifically to hit these latency targets reliably under real concurrent call load, not just in single-call demos. This latency work is a core part of building the full voice agent and connects to selecting the underlying speech-to-text and text-to-speech models.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.