← back
The Base Model Is Dead — Varun Singh, Arcee AI
Takeaway
Design pretraining mixtures as a foundation for later reasoning and agent learning, including task-shaped data from the start.
Summary
- Arcee argues that base-model pretraining should prepare models for downstream reasoning and agentic reinforcement learning rather than mainly reproduce broad web knowledge.
- Recent training recipes reduce general web text and emphasize code and STEM, while differing on whether to include synthetic data.
- Bringing instruction and post-training-style data into pretraining exposes task and conversation structures early instead of reserving them for a final adaptation stage.
- Synthetic rephrasing in models such as Trinity Large and Kimi K2 offers multiple presentations of seed information and can improve quality and task alignment when curated carefully.
pretrainingsynthetic-datareinforcement-learning
Original description
The old story is that a base model is a mirror of the internet, a good model of human web text that everything else gets bolted onto. Varun Singh, who leads pre-training at Arcee AI, argues that story is dead: no modern base model reflects the web the way GPT-3 once did. Instruction data and synthetic reasoning traces have moved earlier and earlier into training, and a distinct mid-training stage has emerged for longer datapoints that look much more like the downstream capabilities you actually want. Reading recent open recipes, from Nemotron to Kimi K2, the pattern is clear: raw web text is taking a backseat. The rest of the talk is what that shift does to how you build. Once reinforcement learning became the thing that got models to reason, the base model stopped being a cherry on top and started needing to carry the prior that RL builds on, which changes the data mix and pulls post-training-flavored data forward. Singh walks through the practical pitfalls his team hit training the Trinity series, like getting the balancing coefficients right and establishing stable representations early so the model is prepared for what it must compose during RL. The message is that as capabilities advance, the base model's job keeps redefining itself, and pretending it still just mirrors the internet will cost you. Speaker info: https://x.com/stochasticchasm / varun-singh-cs Timestamps: 0:00 - The base model as a mirror of the web 1:26 - How knowledge accumulates in training 2:49 - When instruction data moves earlier 4:11 - After o1: RL and reasoning 5:41 - What prior the base model must carry 6:18 - Filtering web text, adding synthetic 8:01 - Reading the open data recipes 9:41 - Lessons from training Trinity 12:02 - Balancing coefficients and early stability 13:30 - Why RL keeps raising the stakes 15:55 - The base model's shifting job