← back
DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve
Takeaway
Evaluate coding agents with original, broadly distributed tasks and leakage-resistant environments, then inspect behaviors behind aggregate scores.
Summary
- DeepSWE contains 113 original long-horizon engineering tasks across nearly 100 repositories and five languages, reducing dependence on mined public pull requests.
- The benchmark targets contamination, solution-specific verifiers, and leakage through Git history; version 1.1 adds protections against recovering golden patches.
- Reported failure analysis distinguishes incomplete multi-part implementations, literal requirement adherence, and models' tendencies to inspect their environment.
- Prompt wording can suppress self-testing even in strong models; DeepSWE leaves test-writing decisions open to reveal those behavioral differences.
deepswecoding-benchmarkscontamination
Original description
DeepSWE is 113 software engineering tasks written from scratch, not scraped from pull requests, so a model cannot have seen them in training. Each one is a long horizon problem drawn from a real open source repository, authored by engineers who actually maintain that code, with isolated environments and program based verifiers that check observable behavior rather than trusting the model's own account. James Shi's point is that once you remove the contamination the leaderboard stops clustering: strong models pull far ahead and others, Gemini 3.1 Pro among them, fall toward the bottom. The more revealing signal is in how models fail. Some quietly expand a task beyond what was asked, a failure mode DeepSWE scores in its own right, and Claude models did this a good fraction of the time while GPT models did it less often. Stronger models also tend not to verify their own work, and there is a real gap between the ones that test what they wrote and the ones that assume it is correct. Since reward hacking is a constant temptation, the verifiers are built to be gamed as little as possible, keeping the score anchored to the objective rather than to a convincing looking rollout. Speaker info: https://x.com/shiqyy / jamesshi117 https://deepswe.datacurve.ai Timestamps: 0:00 - Introduction: the DeepSWE benchmark 1:03 - 113 original, contamination-resistant tasks 2:08 - What makes a good benchmark 3:51 - The leaderboard and model spread 5:18 - Failure mode: over-scoping the task 7:16 - Do models verify their own work? 8:45 - Tasks authored by core contributors 10:15 - Writing realistic, high level prompts 11:45 - Program based verifiers and observable behavior 13:43 - Limitations and future work 15:25 - Reward hacking and keeping it cheating proof