← back
Evals Are Broken, Use Them Anyway — Ara Khan, Cline
Takeaway
Use evaluations as imperfect engineering evidence, combining task relevance with real-world observation.
Summary
- Khan rejects both blind faith in benchmark rankings and model selection based entirely on subjective taste.
- Published lab scores should be treated as approximations; similar aggregate scores can conceal major behavioral differences.
- He recommends allowing new model releases to settle before switching and seeking recent evaluations that precisely match the intended work.
- Cline’s experience illustrates the difficulty of evaluating coding agents when standardized tasks poorly approximate real software engineering.
evalscoding-agentsbenchmarks
Original description
Cline started at 43% on Terminal Bench. The improvements came from container CPU and memory settings, raised timeouts, and prompt engineering techniques specific to Anthropic model families that do not transfer to Codex or Gemini. Not from switching to a better model. Ara Khan's argument is that benchmark numbers are not gospel and vibes are not a system, and that the truth is inconveniently in between. The practical framework: after a run, portfolio allocate the failures by sending another agent through all the failure traces to find which small levers actually move the score. Zone one is obvious bugs. Zone two is the nuance improvements that explain why a model everyone calls great somehow does not work for your specific harness. Zone three is overfitting to the benchmark, which people do, and which Ara is explicitly telling you not to do. Speaker info: https://x.com/arafatkatze / arafatkatze https://github.com/arafatkatze