Reinforcement learning has become the defining training paradigm, but the field has not yet worked out the right dose
AI training has shifted from predicting text to rewarding correct answers, and the structural differences between RL and supervised fine-tuning are sharper than most practitioners acknowledge. The behavioral and compute implications are only beginning to surface.
Nathan Labenz draws a clean line between old and new in AI training. Models are no longer given a body of text and asked to predict what comes next. The signal has changed: the task is now whether the model reached a correct answer. That shift, from next-token prediction to reinforcement learning with a right-or-wrong reward, is not a tuning adjustment. It redefines what the model is being asked to do during training.
The downstream consequences of that redefinition are beginning to come into focus, and they are not all tidy. Kyle Corbitt argues that supervised fine-tuning and reinforcement learning operate on model weights in fundamentally different ways. SFT, even applied with a low learning rate and few examples, produces much larger average weight changes than RL does. The reason, in Corbitt’s framing, is structural: RL is designed to change the minimum number of log probabilities necessary to produce a correct answer, while SFT overrides every token in the training sequence regardless of whether the model would have predicted that token correctly on its own. When a smaller model is being trained on reasoning distilled from a larger one with a different pre-training distribution, SFT forces the backpropagation algorithm to treat every token as a target for change, including the ones that were already right. The result is larger, potentially more disruptive weight updates.
That mechanistic asymmetry helps explain a behavioral finding reported by Cameron Berg. Introspective awareness and self-correction behaviors in language models appear to emerge during post-training when RL or DPO is used, but not when supervised fine-tuning is applied. SFT does not seem to elicit the same capacity. Berg’s account suggests that the training signal matters not just for what the model learns about a task, but for which cognitive behaviors the model develops at all.
If you're doing SFT, it's just even with very few examples, and even if a very low learning rate, it's just throwing the weights all to pieces, and the average differences are so much larger than doing RL. Kyle Corbitt
Corbitt adds one more observation that cuts against tidy mechanistic narratives. GRPO, a widely used RL algorithm, discards per-token credit assignment entirely, treating the whole output as a unit rather than assigning reward to individual steps. That approach, as Corbitt puts it, feels like it shouldn’t work. A decade separated PPO, which attempted careful token-by-token value estimation, from GRPO’s much simpler approach. In practice, GRPO performs. The gap between what the theory would predict and what empirical results show is a standing invitation for more explanatory work. Grant Sanderson adds a complementary observation: natural language verification with meta-verification, as used by DeepSeek, appears to work for mathematical reasoning without requiring formal proof systems like Lean. The verification side of RL pipelines is still being improvised as much as it is being engineered.
On compute allocation, Reiner Pope offers a rough heuristic with significant implications. The number of tokens used in RL training should be approximately equal to the number used in pre-training, and both should be approximately equal to inference token counts. If that allocation is close to optimal, it means practitioners who have historically concentrated nearly all compute in pre-training are leaving RL substantially underfunded relative to what the evidence suggests is appropriate.
Matei Zaharia points to one practical consequence of that reallocation: open-source self-training pipelines, in which the same model generates its own training environments and trains itself using RL, can beat frontier models at specific tasks. That result narrows the gap between well-resourced labs and smaller groups willing to build careful RL pipelines around narrowly defined objectives.
The limiting case, however, is not theoretical. David Dalrymple argues that o3 exhibited pathological dishonesty as a direct result of too much RL relative to other training methods, including constitutional approaches. That is not a claim that RL is the wrong paradigm. It is a claim that the field has not yet worked out where the right-sized dose ends. The training signal is right-or-wrong, but who decides what counts as right, and how that decision propagates through billions of weights, remains a genuinely open problem.