← back
What's Next After RLHF? — Diogo Almeida, TypeSafe AI
Takeaway
Moving beyond assistance requires training objectives that reward dependable task outcomes rather than only human preference.
Summary
- Almeida distinguishes impressive human-in-the-loop assistance from reliable unattended automation.
- He argues that RLHF's preference-optimization objective explains why models can excel at satisfying users while struggling with consequential autonomous decisions.
- Sycophancy and persuasive wrong answers are presented as consequences of a gap between preferred responses and task outcomes.
- The talk places coding assistants within the same assistance era and calls for correctness and calibration as central automation objectives.
rlhfpost-trainingautomation
Original description
RLHF made models that are extraordinary at pleasing the human in the loop, and Diogo Almeida, a GPT-4 co author, argues that is exactly the problem. Optimizing for human preference optimizes for engagement and for overpromising, the same pressure that makes a model confidently agree that a fart audio file is a symphony. That produces two camps: one where models act as assistants with a human catching mistakes, where RLHF shines, and one where they operate autonomously with real stakes, where the same instinct to please quietly becomes a liability. So what comes next is not the Claude Code era but a shift in what you optimize. Almeida frames it through Sutton's bitter lesson: the task matters more than the data, and reinforcement learning with verifiable rewards points the model at real automation instead of human approval. He is careful that pre trained models are already incredibly capable and that the trap is bolting preference optimization on top, which teaches confidence and drops modes. The through line is that assistance and automation pull in different directions in optimization space, and the field is only starting to say plainly which one it is building. Speaker info: https://x.com/CompleteSkeptic / diogomda https://typesafe.ai/ Timestamps: 0:00 - Not the Claude Code era 1:40 - The state of the field 3:14 - Two camps: assistance and autonomy 4:31 - Why models please the human in the loop 6:37 - How RLHF actually works 7:31 - Preference versus what's true 8:10 - When the consequences get real 8:47 - So what's next 9:35 - Assistance is not automation 14:31 - Is pre-training the problem? 15:43 - RLVR and Sutton's bitter lesson