← back
We Cut 94% of AI Coding Tokens With a Local Code Index - Rajkumar Sakthivel, Tesco
Takeaway
Reduce irrelevant input context with measured retrieval, but validate savings and recall against realistic baselines and your own codebase.
Summary
- A local code index chunks functions, classes, and methods, combines semantic and keyword search, and follows call relationships to retrieve focused context.
- Optional compression retains signatures and descriptions; a lightweight relevance formula filters results without an additional model call.
- On 20 questions over a 53-file FastAPI project, the speaker reports reducing context from about 83,000 to 4,900 tokens, with roughly 90% code-retrieval accuracy.
- The claimed 94% saving uses full-file reading as the baseline; real coding-agent savings are lower, and recall nearly collapsed on a larger mixed-purpose codebase.
- A shared index and persistent project memory serve multiple coding tools, reducing repeated discovery across sessions.
code-retrievaltoken-optimizationlocal-index
Original description
Every AI coding tool we tried had the same assumption: send as much context as possible. In our production codebase, that meant sending 45,000 tokens per query — even when only ~5,000 were actually useful. We didn’t notice how inefficient this was until we saw the cost and latency impact. We tried improving prompts and tweaking model settings, but nothing addressed the core problem: we were optimising the model, not the context. So we built a local retrieval layer between the codebase and the agent. Instead of sending full files, we: Structured code using AST-aware chunks (tree-sitter) Combined vector search with keyword matching for better retrieval Used a lightweight relationship layer to follow execution across files The result: 👉 94% reduction in tokens 👉 faster responses 👉 more accurate outputs The hardest problem wasn’t retrieval — it was knowing when retrieval was wrong. We experimented with LLM-based scoring and threshold tuning, but a simple heuristic ended up working best. Everything runs locally, with no data leaving the machine, and one index supports multiple AI tools. In this talk, I’ll walk through: What we got wrong initially Why context matters more than model tuning The architecture behind the system Real benchmarks and trade-offs The key takeaway: 👉 The biggest optimisation in AI coding isn’t the model — it’s the context. Speakers: Rajkumar Sakthivel (Tesco): Rajkumar Sakthivel builds LLM infrastructure at scale and co-created Code Context Engine after his team's AI coding bill jumped from £15 to £200 in a single month. X/Twitter: https://x.com/rajkumarsakthi LinkedIn: / rajkumar-sakthivel GitHub: https://github.com/rajkumarsakthivel