← back
From Transcription to Live Music: Gemini's Audio Stack — Thor Schaeff, Google DeepMind
Takeaway
Audio-native models can combine rich understanding and controllable speech in application workflows beyond plain transcription.
Summary
- Gemini audio understanding extends transcription with speaker labels, timestamps, language identification, translation, emotion, and summaries.
- The Echo Script demo returns these fields through one structured-output API request, while also showing imperfect multilingual recognition.
- Speech generation uses a small set of base voices directed through natural-language scene, accent, and performance instructions.
- The overview connects foundational audio understanding to specialized speech generation and real-time multimodal conversation models.
geminiaudio-understandingtext-to-speech
Original description
One API call to Gemini 3 Flash Preview: speaker labels by name, timestamps, emotion tags, language detection with English translation, and a full summary. That is the audio understanding layer that underlies everything else Thor Schaeff demos here, including speech generation directed by a "director's note" rather than picked from a catalogue, and Gemini 3.1 Flash Live, a sound to sound real time multimodal model with thinking baked in rather than cascaded through a separate LLM. The talk ends with Lyria 3, Google DeepMind's music generation model that can now produce full songs with lyrics. The live demo has the Gemini Live model calling Lyria via tool use on request to generate a German techno schlager about the UK startup scene, live on stage. Speaker info: https://x.com/thorwebdev / thorwebdev