← back
Benchmarking semantic code retrieval on Claude Code — Kuba Rogut, Turbopuffer
Takeaway
Benchmark semantic code search on retrieval quality and workflow costs, since more focused context can improve precision while reducing broad exploration.
Summary
- Rogut treats embeddings as cached semantic computation that can reduce repeated codebase exploration across agent sessions.
- Turbo Grep parses and chunks code, embeds it with a Voyage code model, and stores the index in Turbopuffer for semantic retrieval from Claude Code.
- The experiment uses ContextBench's human-labeled relevant files, lines, and symbols to compare default Claude Code, 50-line windowed reads, and windowed reads plus semantic search.
- Reported file precision rises from about 65% at baseline to 87% with windowing and semantic retrieval, while baseline exploration retains stronger file recall, illustrating a precision-recall tradeoff.
semantic-searchcode-retrievalturbopuffer
Original description
By default, Claude Code wastes one in every three file reads. Add windowed grep and that drops to one in five. Add semantic search on top and it drops to one in eight, with file precision climbing from 65% to 87%. Kuba Rogut from Turbopuffer ran a 50-task benchmark against ContextBench to measure not whether the agent solved the problem but whether it found the right files, lines, and symbols along the way. The benchmark tested three conditions: raw Claude Code, windowed reads capped at 50 lines, and windowed reads plus a semantic search tool backed by Turbopuffer. Semantic search won on behavior adjacent tasks where files share no keywords. Grep won on import tracing where the keyword is right there. Cursor's production numbers show a 24% relative improvement in answer accuracy from semantic retrieval, plus a 2.6% increase in code retention in large codebases. Kuba's explanation for why his gains were smaller: Cursor's model knows when and why to call semantic search. Claude Code just has it as another tool in the list. Speaker info: / kubarogut https://rogutkuba.com/