← back
Designing Voice Agents for Real Conversations - Chintan Agrawal & Daniel Wirjo, AWS
Takeaway
Natural voice conversations require deliberate turn detection and fast interruption cancellation in addition to a capable language model.
Summary
- Voice-agent turn taking depends on audio-pipeline behavior: identical models and prompts can feel radically different when interruption handling changes from nearly two seconds to under 200 milliseconds.
- The architecture combines speech recognition, an LLM, and speech synthesis with voice activity detection, smarter turn detection, and an interruption handler that cancels generation and flushes downstream audio.
- Silero VAD offers lightweight local speech detection, but its silence threshold trades premature interruptions against dead air and cannot distinguish a completed thought from a thinking pause.
- Managed turn detection inside services such as Cartesia and Deepgram uses richer audio and linguistic signals, with the trade-off of limited visibility into incorrect endpoint decisions.
- A third approach keeps local VAD and adds a dedicated Smart Turn model during silence, giving developers a separate turn-completion decision beyond basic speech detection.
turn-takingvoice-activity-detectionaudio-pipelines
Original description
Chat agents get seconds to respond. Voice agents get 200 milliseconds, and if they get it wrong, the user doesn't retry, they hang up. The gap between "impressive voice demo" and "agent people actually want to talk to" is entirely in the engineering: latency budgets, barge-in handling, turn-taking, and the silence detection problem nobody warns you about. In this workshop you'll build a production voice agent using Pipecat (open-source real-time AI framework) that handles the three things most voice implementations get wrong. Interruption: user starts talking mid-response. Do you stop, fade, or finish? The answer depends on context, and we'll build the decision logic. Silence: is the user done talking or just thinking? Get this wrong and you either interrupt them or leave dead air. Latency: you have 200ms total for STT to LLM to TTS. We'll build the streaming pipeline that hits that budget consistently. The thing nobody tells you about voice agents: the hardest problem isn't the AI, it's the audio engineering. WebSocket orchestration, audio chunking, backpressure when the LLM is slower than real-time, echo cancellation when the agent's own output feeds back into the mic. We'll solve all of it. Stack: Pipecat + Daily (WebRTC) + any STT/TTS (Deepgram, AssemblyAI, Whisper) + any LLM. Bring headphones. You'll be talking to your agent by the end. Speakers: Chintan Agrawal (Amazon Web Service): Chintan Agrawal and Daniel Wirjo are Solutions Architects at AWS, focused on AI and SaaS startups. They are passionate about helping customers unlock value from their data and accelerate innovation on AWS. By partnering closely with founders and engineering leaders, they enable organizations to build scalable solutions, drive business growth, and realize the full potential of cloud and AI technologies. LinkedIn: / chintan-agrawal-87a866135 GitHub: https://github.com/wirjo/pipecat-turn...