← back
The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica
Takeaway
Retrieve a small relevant tool set on demand instead of making every request carry the full catalog.
Summary
- Loading every tool schema into every request increases prompt size, tool confusion, latency, and cost as catalogs grow.
- The presented comparison reports accuracy falling from roughly 78% with 10 tools to 13.6% with 741, while semantic routing stays above 83%.
- The proposed design embeds tool descriptions offline, retrieves three to five relevant tools for each query, and injects their schemas just before inference.
- For the 741-tool example, relevant-schema loading cuts about 127,000 tool-context tokens to roughly 1,000; the speakers caution that small catalogs may not justify a router.
tool-routingcontext-engineeringsemantic-search
Original description
The common “Fat Agent” architecture loads a large catalog of tools directly into the system prompt. This often creates latency, cost, and reliability problems in production agent systems. As tool schemas take up more of the context window, agents can become slower and more likely to choose the wrong tool. This session takes a practical look at the Semantic Tool Router pattern, a deterministic layer that reduces the amount of context shown to the model in real time. The talk will share benchmarks across frontier models, including GPT-4o and Gemini 2.0, showing how the number of available tools affects Time-to-First-Token latency and tool-selection accuracy. Attendees will learn how to move from static tool loading to Just-in-Time Context Injection, where only the most relevant tools are added to the prompt for each request. In high-tool-density benchmark scenarios, this approach can reduce response latency by up to 90%, reduce cross-tool confusion, and improve agent reliability. The session will end with a practical framework for building tool routers that can scale to hundreds of capabilities without sacrificing speed or predictability. Speakers: Sohail Shaikh (Prosodica): Sohail Shaikh is a data scientist with nearly a decade of experience across AI, data science, analytics, marketing, and software-oriented work, focused on building practical, reliable, and scalable AI systems using NLP, RAG, conversational intelligence, and LLM workflows. LinkedIn: / sohail-shaikh GitHub: https://github.com/Sohail-Sh Ankush Rastogi (Prosodica): Ankush Rastogi is a Senior Data Solutions Engineer with over a decade of experience building scalable data, analytics, and machine learning platforms, with a focus on turning AI models into reliable, production-ready enterprise systems through strong evaluation, inference performance, cost optimization, and operational design. LinkedIn: / ankushrastogi GitHub: https://github.com/ankushrastogi04