← back
Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse
Takeaway
Improve the definition of success and its domain grounding before spending more tokens optimizing against it.
Summary
- Self-improvement loops need domain-informed objectives because many real-world tasks lack a clear binary success signal.
- A paper-classification experiment used 200 fit examples, 100 validation examples, and 300 held-out test examples.
- A frontier optimizer analyzed error clusters and revised a small model’s prompt, accepting updates only when validation performance improved.
- Accuracy rose from 68% to about 83% before plateauing, with 80.2% on the test set; ambiguous author-selected labels limited the apparently objective target.
langfuseprompt-optimizationdomain-expertise
Original description
We ran auto-improvement loops on a paper classification task against a ground-truth dataset. A real problem, narrow enough to measure precisely, and in fact one of the few clear cut target functions out there. We’ll share how to properly set up an agent for auto-improvement, what task specificity and target function quality is actually required for it to work, and why the most efficient path to a continuously improving agentic system is one where domain experts and automation know when to hand off to each other. Speakers: Annabell Schäfer (Langfuse): Annabell is a Growth Engineer at Langfuse, the largest Open Source AI observability and evaluation platform. She is passionate about building cutting edge AI systems and has been around in the space since 2022. X/Twitter: https://x.com/annabellschfr LinkedIn: / annabell-schaefer GitHub: https://github.com/annabellscha Timestamps: 0:00 Introduction and the goal of avoiding token waste 0:24 The current trend of "designing loops" vs. prompt engineering 0:47 Why coding (with its clear "compile" target) set a false precedent 1:20 Challenges of defining target functions in non-coding domains 2:30 Case study: The arXiv paper classification experiment 3:30 Setup of the minimal self-optimization loop 5:48 The step-by-step iteration process and stopping criteria 7:00 Results: Achieving a 15% improvement in accuracy 9:53 Deep dive into the reasoning behind the improvement 11:26 Translating binary "high signal" feedback to other applications 13:12 Defining what "good" looks like for your specific domain 15:05 The importance of human-agent collaboration and data review Viral Quotes & Potential Titles: "Stop burning your tokens and start building in domain expertise early in your loop design." (0:10) "Teams who are investing heavily in the target function are the ones who manage to continuously improve." (2:02) "If you can't do 'code compiles', you need to wrap your head a little bit differently around what is good and what is not." (14:23) "Don't review it only with your coding agents, but review it as a human." (15:57)