AI model training has shifted from next-token prediction to reinforcement learning with right/wrong answer signals.

Training AIs to find bugs in training pipelines is one of the easier tasks to train for, because most such bugs can be demonstrated with relatively little compute, making it a highly verifiable RL domain.

“My sense is that training AIs to find bugs is going to be one of the easier tasks to train AIs on, because most of these bugs we're talking about can probably be demonstrated without that much compute.”

Goodfire has successfully applied reward shaping techniques to models up to trillion-parameter scale.

“We have succeeded in doing these reward shaping techniques up to the trillion parameter model size.”

Reinforcement learning training time is the main wall-clock bottleneck at Poolside because batch size constraints prevent scaling by adding more GPUs.

“My biggest wall clock bottleneck right now is RL time, right? And it's just because I can't scale it up further because I can't add more GPUs to it because of that bad size constraint.”

o3 was a pathological liar because it had too much reinforcement learning relative to constitutional training.

“03 was really a pathological liar and I think it's it just had too much RL compared to other forms of training like constitutional training.”

Natural language verification with meta-verification, as used by DeepSeek, works for mathematical reasoning without requiring formal systems like Lean.

“It's interesting that natural language verification with some sort of meta-verification seems to work so far in the published literature.”

The main locus of AI iteration has shifted from training model weights to iterating on the agent harness and code layer.

“We've moved from iterating on model weights, training model weights to iterating on this harness and this agent layer.”

Three years ago, reinforcement learning was not considered a major driver for LLM capabilities at OpenAI, contrary to the current widely held belief.

“The second was that reinforcement learning was going to be the biggest driver for LLM capabilities. Today very obvious 3 years ago was not an opinion held or direction held at either OpenAI.”

Ali Behrouz argues that models do not need to be pre-trained.

“We are arguing about models that do not need to be pre-trained.”

Reinforcement learning alone cannot achieve superintelligence due to sample inefficiency; a formal verifier is needed in the inference loop, not just during training.

“If you had infinite resources, you could just RL and it would work maybe. But the reality is that you just can't be sample efficient enough or whatever it is to do that. So that you need some sort of verifier in the loop with the inference process rather than because you do have verifiers and like sort of during the training process and you just don't have them during the.”

Reinforcement learning in practice is not done exclusively on verifiable rewards; it also includes RL against self-critique and an empirical mix of techniques.

“I think people have a sense that we are doing RL on verifiable rewards. And I don't think that's been the case for most of 2025 exclusively.”