← back
Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel
Takeaway
Agent training tasks should be difficult for meaningful reasons, with reliable environments that produce useful learning signals.
Summary
- Snorkel studies task quality in containerized Terminal-Bench-style environments as a form of training-data quality for agents.
- Accepted tasks must be achievable, non-trivial, functionally correct, and supported by a reliable environment.
- Compared with rejected tasks, accepted tasks required roughly twice as many tool calls, more output tokens, and had lower pass rates consistent with greater difficulty.
- Failure analysis separates meaningful reasoning failures from broken-environment failures; the research then compares RL training with the same model, compute budget, and task count.
task-qualityrl-trainingsnorkel
Original description
Same model. Same compute. Same number of tasks. Fine-tuning on low quality tasks improved the base model by 1%. Fine-tuning on high quality tasks improved it by 6%. Kobe Crawford from Snorkel ran that experiment on TerminalBench style agentic tasks and got a 5x difference in training uplift from task quality alone. The talk breaks down what separates the two buckets. Accepted tasks averaged twice as many tool calls, lower pass rates, and more output tokens. Genuinely harder problems. More importantly, their failure modes were cleaner: when a model failed on a well specified task, it failed for a real reason. Rejected tasks tended to fail because of mismatches between what was requested and what the tests actually checked, or because the task never gave the model the context needed to satisfy implicit dependencies. Ambiguous specs do not produce harder tasks. They produce noise. Speaker info: / kobie-crawford https://snorkel.ai/author/kobie-crawf...