← back
Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software
Takeaway
Evaluate long-running agents using multiple measures of task difficulty rather than a single hours-or-tokens headline.
Summary
- Long horizon is a relative scale, with human completion time and model trajectory length capturing different aspects of difficulty.
- METR-style time horizons report success thresholds on tasks of a given human duration, rather than literal agent runtime.
- Token counts, steps, and tool calls depend heavily on the model and harness; human timings depend on expertise and methodology.
- Tedious human tasks may be trivial to script, so environment complexity and coordination across tools are additional capability dimensions.
long-horizonagent-benchmarksrl-environments
Original description
Everyone wants agents that handle long horizon work, but Rayan Garg starts with the awkward question of what long horizon even means. One popular answer measures the time horizon as the task length at which an agent crosses a success threshold, like the sixteen hour mark, which is a useful endpoint but a noisy one, since human time estimates vary and the same wall clock hides very different amounts of real difficulty. How you choose to measure this has an outsized effect on what you conclude about a model. From there Theta Software's work is about designing the environments and verifiers that make those measurements honest. A task can be artificially stretched by forcing serial dependencies, or made genuinely hard when a bad early query cascades through everything after it, and as environments grow more complex, standardized evaluation gets harder and correctness is best verified from the final state rather than a judge's guess. Garg walks through collapsing a huge state space with sample trajectories, being careful that judges do not see information they should not, and reusing agents to sift artifacts like CI logs. The recurring principle is that long horizon progress lives or dies on environment and verifier design, not on the headline benchmark number. Speaker info: https://x.com/RayanGarg / rayan-garg Timestamps: 0:00 - What does long horizon mean? 1:13 - Time horizon and the threshold metric 3:17 - Why the metric is noisy 4:20 - Measuring what actually matters 6:38 - Creating tasks and environments 7:42 - When a bad early step cascades 10:01 - Why standardized evaluation is hard 11:17 - Verifying from the final state 13:46 - Judges, tools, and reused agents 17:45 - Rubrics, QA, and careful grading