← back
Beyond Transcription: Building Voice AI That Understands Conversations — Hervé Bredin, pyannoteAI
Takeaway
Conversation understanding requires speaker identity and timing in addition to an accurate sequence of words.
Summary
- Speech-to-text captures words, while speaker-attributed transcription identifies who said them for meeting notes, dubbing, and podcast analysis.
- Precise timing exposes interruptions, overlapping speech, pauses, and short backchannels that can change conversational meaning.
- Prosody, emphasis, nonverbal sounds, addressees, and acoustic context offer additional signals for downstream understanding.
- Pyannote’s diarization pipeline goes beyond voice activity detection and segmentation to assign speaker identities to individual turns.
speaker-diarizationpyannotespeech-understanding
Original description
The open ASR leaderboard reports Nvidia Parakeet at 11.4% word error rate on AMI meeting data. Hervé Bredin runs the same model on the same dataset and gets 26%. Same model, same recordings, different microphone: the leaderboard uses headset audio, he uses the table mic. Most voice AI benchmarks are measuring single speaker speech and calling it solved. The talk covers speaker diarization (who speaks when), why combining it with transcription is harder than it looks, and what breaks at the word level when two speakers overlap. Bredin demos live on a two speaker phone call, walks through the word that falls between two speaker boundaries with no clean owner, and runs pyannoteAI's Precision 2 model down to 3% diarization error against the open source baseline at 5%. State of the art today: 2% on clean telephone calls, 41% in a noisy restaurant. Speaker info: https://x.com/hbredin / herve-bredin https://github.com/hbredin