← back
Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean
Takeaway
Choose and continually evaluate models per request using explicit workload preferences instead of a single leaderboard winner.
Summary
- DigitalOcean frames routing as a way to control inference cost, match model capability to requests, and provide failover when providers degrade.
- Its open-source proxy planner and specialized routing model incorporate task descriptions, user preferences, cost, latency, and model constraints, with reported routing latency below 200 milliseconds.
- Configuration supports ranked fallback models and selection based on recent serving speed, alongside editable decision rules and task presets.
- A demonstrated evaluation reports 90% correctness for a router versus 95% for a single premium model, with lower token use and faster responses, emphasizing evaluation on the user's workload.
model-routinginference-costdigitalocean
Original description
Two terminals run the same prompt, build me a spinning wheel app. On the left every request goes to a single premium model. On the right they go through a router that picks a model per task. Both finish at about the same time with comparable output, and by then the router's session has cost 8 cents against 25. The gap widens with every prompt after that. Archana Kamath and Tyler Gillam use it to argue that picking a model by climbing a leaderboard is the wrong instinct, because there is no single best model, only the right one for a given request. What makes a model right is a mix no public leaderboard encodes: the task itself, the system prompt and tools around it, the cost you are willing to spend, the latency the use case needs, and what the end user actually wants. Their router takes those as preferences you declare, in natural language or as decision tree rules, then honors them per request. It runs on a purpose built mixture of experts model that decides in under 200 milliseconds, costs nothing extra, and is open sourced along with the proxy in front of it. Gillam then shows the part that separates it from a vibe check, an evaluation scoring the router at 90% correctness against 95% for the single premium model while using far fewer tokens and returning faster. Routing is the foundation layer, with evaluation, caching and personalization built on top. Speaker info: / tdgillam Timestamps: 0:00 - Why the one model habit is breaking 2:42 - There is no single best model 4:21 - A router you can customize and evaluate 6:57 - Configuring tasks, model pools and failover 7:48 - Side by side in the playground 9:29 - Proving it with an evaluation 10:18 - Two coding agents, and the session cost gap 13:49 - Under 200ms, open sourced, no code changes 14:43 - Evaluation, caching, personalization