← back
From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google
Takeaway
Choose system AI for optimized integration or app-bundled models for customization, and load local skills only when needed.
Summary
- On-device inference offers privacy, offline operation, lower latency, and potential savings; tiny models are described as smaller than one billion parameters.
- Developers can use system-provided Gemini Nano through AI Core or bundle customized models through LiteRT.
- Google AI Edge Gallery demonstrates local chat, image and audio tasks, third-party models, and agent skills on mobile devices.
- The skill harness puts descriptions in context, loads details on demand, and renders JavaScript interfaces such as maps or restaurant roulette.
on-device-ailitertagent-skills
Original description
Function Gemma ships at 270 million parameters and processes nearly 2,000 tokens per second prefill on a Pixel 7. Out of the box, on a fixed set of app intents, it hits 46% accuracy. Fine-tuned on a synthetically generated dataset, it clears 90% on eight of ten functions. Cormac Brick covers the two options developers have for on-device AI: Gemini Nano via AI core for common tasks, and LiteRT-LM for custom models that ship inside your app. The session walks through a live skill harness built on Gemma 4 with a restaurant roulette demo running fully on-device, and Eloquent, a production transcription app built by chaining two models under a few hundred million parameters. Speaker info: / cbrick Timestamps: 0:00 Introduction to on-device agents and tiny LLMs 0:48 Overview of AI Edge, SLMs, and TLMs 0:57 Taking a look at agent skills 1:06 Taking a look at tiny models 1:24 Motivations for on-device AI (latency, privacy, offline use) 3:01 System-level GenAI (Gemini Nano via AI Core) 4:03 App-level GenAI (LiteRT-LM for custom/boutique models) 5:06 Google AI Edge Gallery app demo 6:22 Deep dive into agent skills and the skill harness 7:41 How the skill harness works (system prompts, tool calls, and JavaScript UI) 9:00 Creating and publishing your own skills 10:28 Using LiteRT-LM runtime for model deployment 12:31 Export and inference workflow (from PyTorch to deployment) 13:19 Function Gemma: Robust, small-scale function calling 14:35 Fine-tuning workflow for tiny models using synthetic data 16:01 Eloquent: A production transcription app example using tiny models 17:28 Q&A: Agent skill robustness and multi-skill calling 19:26 Q&A: LiteRT-LM file format vs. Task files 20:00 Q&A: Performance on CPU/TPU and resources