← back
Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI
Takeaway
Engineer voice quality as an end-to-end streaming system with explicit latency, transcription, turn-taking, and scaling requirements.
Summary
- Production voice agents must jointly achieve low latency, capable tool use, natural speech, and reliability across thousands of concurrent calls.
- The cascading architecture streams audio through speech recognition, an LLM, and speech synthesis, coordinated by frameworks such as Pipecat or LiveKit.
- Speech-recognition errors on names or medications propagate downstream; Bhargava cites roughly 6% benchmark word error rates and Together transcript-completion latency around 100 ms at P90.
- Turn detection must distinguish pauses from completed utterances; streaming-native encoders use limited look-ahead and cached activations to avoid the overhead of adapting Whisper's 30-second batch design.
voice-agentsstreaminglatency
Original description
Users notice latency above 500ms and hang up above one second. In an already optimized pipeline, 75ms of network latency from models sitting in a different data center adds 30% overhead. Colocating everything in the same building drops that to around 5ms. Rishabh Bhargava from Together AI walks through the full speech to text, LLM, and text to speech pipeline at that level of specificity. The LLM dominates the budget: 200 to 300ms time to first token target, 8 to 30B parameter range — larger models blow the latency budget, smaller ones break tool calling. Speech to text target is P90 under 100ms with around 6% word error rate. One pattern for handling complex workflows without adding latency: a small thinker LLM handles conversation flow and issues a single tool call to a larger model when the request is complex, keeping the fast path fast. Speaker info: / bhargavarishabh