← back
Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect
Takeaway
Extending RL to real work requires better feedback and reward design beyond deterministic correctness checks.
Summary
- Brown describes reinforcement learning as updating model weights toward higher rewards from agent-environment interactions.
- Environments combine tasks, worlds, harnesses, and scoring, and can support evaluations, supervised data, distillation, or prompt optimization as well as RL.
- Reports, purchases, and customer interactions often lack a single verifiable answer, making reward design and distribution coverage difficult.
- Loose proxy objectives create reward-hacking risks when models discover ways to increase scores without improving real outcomes.
- The goal is continual learning in messy deployments, with systems detecting mistakes and improving under appropriately placed human supervision.
reinforcement-learningreward-designcontinual-learning
Original description
Reinforcement learning has been easy to sell where the answer is checkable, like math or code, and Will Brown's talk is about everything else. Most valuable tasks have no clean verifier, so Prime Intellect's work is on how you build reward signal when there is no ground truth waiting. He frames RL simply first, a model acting in a harness with tools and skills, getting a reward, and nudging its weights, then asks how you keep climbing once you leave the verifiable island behind. His answer leans on environments as the anchor. You can set up judges, generate question and answer pairs grounded in real documents and repos, and use a reverse direction trick where you hide something, like a bug or a backdoor, so the model can learn to find it again, which conveniently gives you a difficulty dial to keep tasks not too easy and not too hard. He is direct about the dangers: reward hacking will find you if you are not careful, so you inspect traces, run small experiments, and bring in expert understanding. The goal he keeps returning to is making this a real science, with open models and shared benchmarks, where environments turn into new tasks and higher levels of ability. Speaker info: https://x.com/willccbb / willcb https://willcb.com Timestamps: 0:00 - RL without verifiable rewards 1:17 - How RL works, simply 2:43 - The tooling that powers it 4:13 - Where verifiable rewards run out 6:24 - Being careful about reward design 8:04 - Making RL a science 9:20 - Judges and grounded question answer pairs 10:46 - The reverse direction trick 14:19 - Calibrating difficulty 15:08 - Hunting for reward hacks 18:27 - Environments as the anchor