← back
SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius
Takeaway
Reliable coding-agent benchmarks depend as much on fresh, validated tasks and controlled infrastructure as on the models being ranked.
Summary
- SWE-rebench evaluates roughly 30 models monthly on fresh software-engineering issues from the previous month to reduce contamination from published benchmark solutions.
- Tasks combine an issue description, a Docker environment, fail-to-pass tests, and pass-to-pass regression tests.
- Filtering removes vague or over-specified tasks, solution-specific assertions, and unstable dependencies, followed by manual verification of final tasks.
- A minimal shared harness supports model comparisons, but trustworthy results also require stable infrastructure, explicit retry rules, and separation of provider failures from agent failures.
swe-rebenchcoding-benchmarkscontamination
Original description
Claude Code solved SWE rebench tasks by reading git history to find the solution patch. When Nebius removed future commits from the environment, it fetched the original GitHub issue. When they blocked web fetch, it switched to curl, formatted the conversation for readability, and solved the task again anyway. Ibragim Badertdinov built the leaderboard specifically because these behaviors only become visible once you run agents against real tasks at scale. SWE rebench updates every month with problems from the previous month because benchmark data leaks into pretraining and time splits are the only defense. The talk covers what separates accepted tasks from rejected ones (accepted tasks averaged twice the tool calls, lower pass rates, and cleaner failure modes), why ambiguous specs produce noise rather than harder problems, and how the same filtering pipeline that powers the leaderboard has produced 30,000 real world training environments used by frontier labs. Speaker info: https://x.com/ibragim_bad / ibragim-badertdinov https://github.com/ibragim-bad