← back
Your Voice Agent Doesn't Need a Frontier Model - Joel Allou & Ornella Bahidika, Microsoft
Takeaway
A small speaking model backed by explicit lesson logic can meet voice latency budgets more effectively than a frontier model reasoning through every turn.
Summary
- ACE targets speech onset around 950 milliseconds, making multi-second reasoning delays unacceptable for live tutoring.
- Lesson state, student mastery, and next-step logic are handled outside the speaking model and summarized for each turn.
- A demo contrasts a slower Opus 4.7 response with a scaffolded Haiku 4.5 response arriving in roughly 900 milliseconds.
- Smaller models still need strict scaffolding to avoid structural drift; the engineering cost is paid in the harness rather than repeated reasoning on every turn.
voice-agentslatencysmall-models
Original description
My AI voice tutor doesn't run on a frontier model. It runs on a small one, and the reason isn't cost. It's that voice lives or dies on latency, and the scaffolding around the model is what makes it feel smart anyway. When you build a voice agent the clock is brutal. A pause longer than a held breath feels broken, so your real budget is time to first token, not benchmark score. A big model that thinks for a second has already lost the room. So the model choice gets made for you: pick the fastest one the latency budget allows, then make up the intelligence elsewhere. I'll show how that plays out in an AI voice tutor I built on a small, fast model. The model never has to remember what the student knows, plan the lesson, or decide what comes next. Deterministic systems do all of that and hand the model a tight, structured brief each turn. What's left for the model is the one thing it's genuinely best at, which is talking. The scaffolding isn't a cost optimization bolted on afterward. It's the thing that lets you use the cheap fast model at all. Seven minutes. The latency budget that forces the decision, what moves out of the model to survive it, and where a small model still falls down no matter how much scaffolding you give it. Speakers: Joel Allou: Joel builds voice-first AI tutors. Solo founder focused on agentic systems for personalized learning, with a particular interest in infrastructure that makes flow agents reliable. LinkedIn: / joel-allou Ornella Bahidika (Microsoft): Ornella Bahidika is a Product Manager at Microsoft, where she develops solutions that help organizations optimize collaboration, workplace technology, and AI-driven experiences. LinkedIn: / ornella-bahidika