← back
Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI
Takeaway
Start context optimization by removing or shortening low-value historical tool output, then measure the effect on quality, cost, and latency.
Summary
- Towards AI uses an open-source course tutor to study context management, with requirements for course-grounded answers, long debugging sessions, code handling, and low latency.
- The workshop distinguishes within-session context management from cross-session memory and focuses its experiments on the former.
- Accumulated tool outputs and retrieved course chunks inflate repeated prompts, increasing cost and time to first token while potentially degrading answer quality.
- Low-cost compaction strategies include truncating unusually large tool outputs with explicit markers, retaining only recent turns, and clearing selected old tool results while allowing retrieval again when needed.
context-engineeringcompactionai-tutors
Original description
The cheapest configuration they tested was the one sending the most tokens. Across 11 presets run against their open source AI tutor, doing nothing at all to the context beat every compaction technique on recall, cost, and latency at once, and their own production defaults scored worse than leaving the history alone. Prompt caching is why. With 97% of tokens served from cache, and cached tokens up to 50 times cheaper on some APIs, compaction has to shrink a context by more than 50 times before it pays for itself, because rewriting the context invalidates the cache. Louis-François Bouchard's framing is that summarization is potentially a trap. Omar Solano walks through the architecture and the first run, including a knowledge base browsing tool they built, measured, and found returned identical recall while running 50% slower. Samridhi Vaid extends it: keeping the full history recovered specific details 95% of the time against 32% after summarizing, and distinctive facts survived to 800,000 tokens without visible rot. Hardware changes the answer, though. Capped locally at a 32k window, keeping everything stops being possible, and a larger parameter count does not buy a larger context window. Dense retrieval fell to 0% recall on facts buried at 400k tokens where BM25 still found them every time. The rule they land on is to name the constraint you actually have before reaching for compaction, rather than compacting by default. Speaker info: Louis-François Bouchard (Towards AI): https://x.com/Whats_AI / whats-ai https://www.louisbouchard.ai Omar Solano (Towards AI): https://x.com/omar_solano1 / omar-solano1 Samridhi Vaid (Towards AI): https://x.com/samridhivaid / samridhivaid Project: https://github.com/towardsai/ai-tutor... Timestamps: 0:00 - The problem is the context, not the model 1:30 - The AI tutor, and its five requirements 5:21 - Two root problems: a finite window, a stateless model 7:52 - Context rot, cost, and latency 9:13 - The compaction toolkit, with and without an LLM 12:58 - Offloading to files, the LLM wiki, progressive disclosure 16:47 - Prompt caching, and why compaction can backfire 19:20 - When to clear, compact, and optimize for cache hits 21:53 - The tutor's architecture, a single ReAct agent 25:49 - Hybrid search over an 8 million token corpus 28:28 - Letting the agent browse the knowledge base 31:01 - The browse tool measured: same recall, 50% slower 36:23 - The experiment setup: presets, tasks, harness 42:55 - Results: doing nothing wins on all three fronts 48:13 - Should you ever compact? 49:32 - DeepSeek, and a 50 times cache discount 50:53 - Memory: 95% against 32% after summarizing 54:38 - Cost at scale, and going local 57:09 - Local limits: bigger models, same window 58:25 - Where dense retrieval fails and BM25 holds 1:01:01 - What they finally chose