← back
Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning
Takeaway
Scaling reasoning requires the right training objective and data, beyond strong base-model benchmarks.
Summary
- Ross Taylor contrasts Galactica's strong scientific benchmark results with ChatGPT's product success to argue that a capable base model needs effective post-training.
- Galactica used a curated 105-billion-token corpus, repeated training epochs, and internal thinking tags to improve data efficiency and reasoning.
- An unpublished Llama 2 reasoning recipe combined continued pretraining on mathematics and science with PPO, verifiable rewards, and an outcome reward model to initialize the value model.
- That recipe improved math performance but did not produce the reflective behavior and inference-time scaling later associated with reasoning models.
reinforcement-learningreasoningpost-training
Original description
Ross Taylor opens with some history: back in 2022 he worked on Galactica, an early large model for science that briefly crossed the Rubicon on curated high quality data and intermediate reasoning tokens before the reaction overshadowed the work. That obsession, optimizing what happens between the question and the answer, is where this talk on long horizon reinforcement learning picks up. He and Chengxi Taylor of General Reasoning treat long horizon less as a benchmark and more as a mindset: if you want agents that stay coherent over hours, you have to be patient about signal and deliberate about how you spend tokens. The mechanics they walk through are the ones that make long rollouts trainable. Value models reduce variance and help with credit assignment, bootstrapping pulls signal out of sparse rewards, and the real constraint becomes the tradeoff between off policy staleness and GPU utilization as sequences get longer. They make it concrete with a task where frontier models were handed real money to trade football matches and did poorly, exposing how little the environment was actually simulated. The takeaway is that scaling to long horizons demands better environments and simulation, not just bigger context windows, and they point listeners to openreward.ai to go deeper. Speaker info: Ross Taylor (General Reasoning): https://x.com/rosstaylor90 / rosstaylor90 https://rossjtaylor.com Chengxi Taylor (General Reasoning): https://x.com/chengxitaylor / chengxi-taylor https://www.chengxitaylor.com/ Timestamps: 0:00 - Introduction and a look back 1:57 - The Galactica story 5:15 - Curated data and thinking tokens 8:09 - What got RL cooking 9:12 - Long horizon as a mindset 10:16 - Why value models help 11:08 - Credit assignment and bootstrapping 12:38 - Trading football matches for real money 13:44 - Why models struggled 14:36 - Off policy staleness versus GPU use 16:18 - openreward.ai and what's next