What is dpo?

The delta learning hypothesis (Scott Gang) explains that DPO learning signal comes from the contrast between chosen and rejected pairs, not the absolute quality of the chosen response.

“This was the core intuition behind the paper that inspired our DPO work which is the delta learning hypothesis by Scott Gang. And the idea is that in DPO and preference learning, it's the contrast between the pair of chosen and rejected answers rather than the quality of the chosen that impacts the learning signal for your model.”

Introspective awareness in LLMs emerges only from RL-based post-training (e.g., DPO), not from supervised fine-tuning, and not at all from pre-training, explaining why SFT-only models lack the self-correction effect seen under DPO.

“One interesting thing they found is that this capability seems to emerge in post-training, not in pre-training, and that even different methods of post-training, like different RL algorithms, DPO verse well, basically different forms of learning algorithms in post-training. So, like RL algorithms seem to induce this, but supervised fine-tuning, which is a supervised learning, doesn't emerges in this very interestingly idiosyncratic way.”

Nathan Lambert on DPO being underrated despite community resistance.

“I think that DPO is very much this direct preference optimization but optimization is colloquially in communities right now where people just don't want to use it. We found it to be incredible low-hanging fruit and one of the easiest things to get off the ground for our models in return for performance.”

Most open models have saturated in performance, causing standard DPO datasets like UltraFeedback to yield chosen/rejected pairs with too small a quality delta to produce a useful learning signal.

“Most of the open models out there actually are saturating in performance where most of the responses are similar.”

DPO is broadly dismissed in the research community but Nathan Lambert argues it remains low-hanging fruit that stacks gains on top of RL training.

“Doing RL on top of DPO is kind of stacked gains and whenever we revisit it, the models that we trained with RL if we were to just skip DPO tend to be worse.”

Finbarr Timbers on DPO's fading status among practitioners.

“People that use DPO will be looked down upon, but it's like if you're trying to bootstrap a recipe off the ground and just take gains where you can, I still think it'll work for a lot of people.”