What happens during pre-training shapes reasoning after fine-tuning, and the field is starting to prove it

The conventional account of how language models learn to reason places most of the weight on post-training: fine-tuning, reinforcement learning from human feedback, and the various alignment stages that follow a base model’s release. Yejin Choi’s finding cuts against that account in a specific and measurable way. Pre-training with thought tokens, she reports, produces a base model that performs meaningfully better on reasoning tasks even after the full post-training pipeline runs on top of it.

The result holds, Choi notes, under controlled conditions. The comparison is FLOP-matched: the improvement is not an artifact of spending more compute during pre-training and then measuring the output. The model trained with thought tokens simply exits the pre-training stage better positioned, and that advantage survives fine-tuning and reinforcement learning rather than being washed out by them. Choi describes the finding as a genuine surprise, which is itself worth noting: this is a researcher with a strong prior on what pre-training can and cannot do, reporting that the data revised her expectation.

The external record supports the direction of that finding. A body of recent academic work, including a paper available at arxiv.org/html/2510.14865v1 alongside several related preprints, examines systematically how pre-training composition and mid-training data quality affect post-training outcomes. Across controlled settings, the research documents that reasoning-intensive pre-training data, including what some papers describe as mid-training phases specifically designed to surface structured reasoning, translates into measurably better performance after subsequent fine-tuning. The effect is not marginal. It appears consistently enough across independent research groups that it is becoming a reproducible pattern rather than a lab-specific observation.

What we found to my big surprise is that when you finish your pre-training in this way even in the full controlled setting the final pre-trained model does do much better after post training. Yejin Choi

What makes this pattern non-obvious is that post-training is expensive, iterative, and highly optimized. The expectation, reasonable on its face, is that a sufficiently intensive fine-tuning and RL process will converge to similar endpoints regardless of where the base model started. The evidence Choi describes, and that the arxiv literature corroborates, is that this expectation is wrong, or at least wrong enough to matter. The base model’s internal structure at the end of pre-training is not a detail that post-training erases. It is a constraint that post-training works within.

The practical implications point in two directions at once. For labs that treat pre-training as a commodity phase and concentrate their research investment in post-training alignment and capability work, the finding suggests a possible source of systematic underperformance that would be hard to diagnose from post-training metrics alone. A model that exits pre-training without the structured reasoning capacity that thought tokens appear to instill may simply have a lower ceiling, regardless of how much RLHF or supervised fine-tuning follows. That ceiling would show up as an asymptote in post-training benchmarks that looks like a post-training problem but is, at its root, a pre-training one.

For researchers working on pre-training data curation and training objectives, the result is more directly actionable. If thought tokens during pre-training produce a durable improvement that persists through subsequent stages, then pre-training data composition is not just a question of coverage and scale. It is a question of whether the training signal asks the model to reason in ways that compound rather than stall. The benchmarks now in the literature are specific enough that the claim can be tested by teams working independently of Choi’s group, and the arxiv preprints suggest that process is already underway.

The deeper question the finding opens is structural. Pre-training and post-training are typically designed and staffed as separate problems, with different teams, different evaluation frameworks, and different optimization targets. If the quality of reasoning that emerges from post-training is in fact downstream of choices made during pre-training, that division of labor may be producing blind spots that no amount of optimization within either stage can fully resolve. Whether the field reorganizes around that insight or absorbs it incrementally into existing pipelines is, at this point, an open question. The evidence, at minimum, makes ignoring it harder to justify.