Quiz¶
From the video https://www.youtube.com/watch?v=v5mBjeX4TJ8 and blog https://aimlbling-about.ninerealmlabs.com/blog/benchmarks-predict-model-size/
Using LLM Memorization Capacity to Estimate Number of Parameters
The blog uses benchmark accuracy as a proxy for memorization capacity, and regresses it against the total number of parameters to estimate the size of closed frontier models.
In our lectures, we saw that what drives memorization in a neural network is not simply the total parameter count. For this regression to be valid across frontier models, an additional assumption must hold.
Which assumption is required?
A. All frontier models are MoEs, and they all have approximately the same depth (i.e., the same number of transformer blocks), so total parameter count scales uniformly with model capacity.
B. All frontier models are MoEs, and they are all trained on approximately the same data, so they memorize the same things.
C. All frontier models are MoEs, and the ratio of total parameters to active parameters ratio is approximately the same across all models, so total parameter count is proportional to the active parameters that actually drive memorization.
D. All frontier models are MoEs, and they all use the same attention mechanism (e.g., multi-head attention with the same number of heads), so the representational capacity per parameter is constant across architectures.
Answer Key¶
Correct answer: C
| Option | Assessment | Reasoning |
|---|---|---|
| A | ❌ Wrong | Depth affects optimization and model smoothness more than it affects how the function fits the data. |
| B | ❌ Wrong | Data affects what is memorized, but we can probe for content the models were all trained on. |
| C | ✅ Correct | In MoEs, memorization is governed primarily by the hidden-layer dimension of the MLPs rather than by the total parameter count. Adding experts helps, but a wider expert MLP helps more, because only the routed experts process any given token and width is the key to better fitting. For total parameters to serve as a valid regression proxy, the sparsity ratio (total / active) must be roughly constant across models, so that memorization can be a proxy for model size. |
| D | ❌ Wrong | Attention has very few parameters and gives the model the ability to reason over text, but it has little to do with capacity to memorize — that is the MLP's job. |
We know how to train LLMs with next-token prediction on large text corpora. The result is a model that can complete text — but not necessarily one that follows instructions, avoids harm, or behaves helpfully.
How do we align an LLM to human preferences after pre-training?
For well-defined tasks (math, code), correctness is checkable.
For open-ended tasks — writing, summarization, instruction-following — there is no ground-truth checker.
We need humans to evaluate outputs. But humans cannot reliably assign scalar scores; they are far more consistent at relative judgments.
Today's topic. Preference optimization methods differ in what statistical model they assume for human feedback — not in whether they use RL. This single axis unifies RLHF, DPO, IPO, and KTO, and clarifies what the KL term is doing in all of them.
Roadmap (we are unlikely to finish)¶
- The feedback: why pairwise comparisons, not scalar ratings
- Bradley-Terry model: the latent-utility noise model behind preference learning
- RLHF — Stage 1: reward model; Stage 2: RL policy optimization
- The LLM as an RL policy
- Policy gradient for LLMs
- KL-regularized objective and its closed-form solution
- PPO in practice
- DPO: collapsing two stages into one loss; what is and isn't "direct"
- When DPO breaks: near-deterministic preferences, the KL collapse, and IPO
- Beyond pairwise data: KTO and unpaired binary feedback
- The contrastive lens: unifying all methods
Running example (used throughout)¶
Setting. An LLM answers medical questions. A physician compares two candidate responses and marks which is clearer, more accurate, and safer.
Question: "What are the risks of combining warfarin and ibuprofen?"
| Response | |
|---|---|
| $y^+$ (chosen) | "Combining these significantly increases bleeding risk. Ibuprofen inhibits platelet aggregation and displaces warfarin from plasma proteins, raising its free concentration. Monitor INR closely and consider alternatives such as acetaminophen." |
| $y^-$ (rejected) | "These medications may interact. Consult your doctor before taking them together." |
The physician compares the two: they prefer one over the other.
We only observe: $(x,\, y^+,\, y^-)$. Can we learn a reward model and then a policy from this data alone?
1. Why pairwise comparisons?¶
Why not ask for scalar ratings (1–10)?
Humans are inconsistent at assigning absolute scores. The same output rated by the same person at different times will receive different scores:
- inter-rater and intra-rater agreement collapse on absolute scales.
But humans are far more reliable at relative judgments:
- "this one is better than that one." Psychophysics has known this since Thurstone (1927). Relative comparison activates a more robust cognitive process than absolute rating.
Statistical implication. The implication is:
$$\text{"} y^+ \succ y^- \mid x \text{"} \quad \text{("$y^+$ is preferred over $y^-$ given context $x$")}$$
with some probability $P(y^+ \succ y^- \mid x)$ that models human decision noise.
Every modeling choice downstream: reward model, policy objective, regularizer, is designed based on how we model this noise.
Key design question. What is the right noise model for the pairwise comparison probability?
2. The Bradley-Terry model¶
The preferred choice in preference optimization is Bradley-Terry (1952): preferences are generated by a latent reward (or utility) $r(x, y) \in \mathbb{R}$ through a logistic link:
$$\boxed{\,P(y^+ \succ y^- \mid x) \;=\; \sigma\!\bigl(r(x, y^+) - r(x, y^-)\bigr) \;=\; \frac{e^{r(x,\, y^+)}}{e^{r(x,\, y^+)} + e^{r(x,\, y^-)}}\,}$$
where $\sigma(z) = \frac{1}{1+e^{-z}} = \frac{e^z}{1+e^z}$ is the logistic sigmoid.
- Interpretation. The preference probability depends only on the difference in latent utilities, not on their absolute values.
- Consequently, $r(\cdot, x)$ is identifiable only up to an additive constant per context $x$ — a property DPO will exploit later.
Why the logistic link? Bradley-Terry is equivalent to assuming the latent utilities are corrupted by independent Gumbel noise and the human picks the higher noisy utility (Yellott, 1977).
- It is the most common choice when one only constrains $P$ to depend on the reward difference.
In our example:
$$P(\text{physician prefers detailed response}) \;=\; \sigma\!\bigl(r(x,\, y^+_\text{detailed}) - r(x,\, y^-_\text{vague})\bigr)$$
- A large reward gap $\to$ near-certain preference.
- A gap near zero $\to$ near-chance comparison.
Other noise models exist (Plackett-Luce for ranked lists, Thurstone's Case V for Gaussian noise), but Bradley-Terry is the foundation of most LLM preference work.
3. RLHF: Overview¶
Reinforcement Learning from Human Feedback (Christiano et al., NeurIPS 2017; Ziegler et al., 2019; Ouyang et al. InstructGPT, NeurIPS 2022) is a two-stage pipeline:
$$\underbrace{\mathcal{D}_\text{pref} = \{(x_i, y_i^+, y_i^-)\} \;\longrightarrow\; \hat{r}_\phi}_{\textbf{Stage 1: fit reward model}} \quad \longrightarrow \quad \underbrace{\hat{r}_\phi \;\longrightarrow\; \pi^*_\theta}_{\textbf{Stage 2: RL policy optimization}}$$
- Each stage introduces its own assumptions and failure modes. Understanding both stages is essential for understanding every method that came after RLHF: including DPO, IPO, and KTO.
3a. Training the reward model in practice¶
Starting point: a pretrained LLM with a language-model head.
Input tokens → Transformer backbone → LM head → logits over vocab (size V)
Goal: a scalar reward $r_\phi(x, y) \in \mathbb{R}$ for any prompt–response pair.
Two architectural changes:
- Replace the LM head with a linear projection to a single scalar.
- Format the input as the concatenation $[x;\, y]$ — prompt followed by response.
[x ; y] tokens → Transformer backbone → final hidden state h → w^T h → r_\phi(x,y) ∈ ℝ
(last token, dim d) (dim d→1)
The backbone weights are initialized from the pretrained LLM and fine-tuned. The scalar head $w \in \mathbb{R}^d$ is initialized randomly and trained from scratch.
Architecture. Take a pretrained LLM (typically the SFT checkpoint) and replace the LM head with a scalar projection $w \in \mathbb R^d$ on the last-token hidden state:
$$r_\phi(x, y) \;=\; w^\top\, h_\phi([x;\, y])_\text{last}.$$
The transformer backbone is initialized from the LLM and fine-tuned; $w$ is trained from scratch.
Loss. Maximum likelihood under Bradley-Terry on a preference dataset $\mathcal D_\text{pref} = \{(x_i, y_i^+, y_i^-)\}$:
$$\mathcal{L}_\text{BT}(\phi) \;=\; -\,\mathbb{E}_{(x,\, y^+,\, y^-)\sim\mathcal D}\!\bigl[\,\log \sigma\bigl(r_\phi(x, y^+) - r_\phi(x, y^-)\bigr)\,\bigr]$$
This is binary cross-entropy on the reward gap: $[x;y^+]$ and $[x;y^-]$ are forwarded through the same $\phi$, and the gap is pushed in the direction of human preference.
Identifiability. $\mathcal L_\text{BT}$ is invariant to adding any context-dependent constant $c(x)$ to all rewards. The reward model identifies $r_\phi$ only up to additive shifts per context: only differences are meaningful. (DPO will exploit this.)
Practical notes. Reward models overfit fast: typically 1 epoch with AdamW, learning rate $10^{-6}$–$10^{-5}$. Hold out 5–10% of pairs and monitor pairwise accuracy.
3b. RLHF Stage 2: the LLM as a sequence-level bandit¶
The LLM is already a policy. An autoregressive LLM defines a conditional distribution over completions given a prompt:
$$\pi_\theta(y \mid x) \;=\; \prod_{t=1}^{T} \pi_\theta(y_t \mid x,\, y_{<t}).$$
No architectural change — the policy parameters $\theta$ are the LLM weights.
Sequence-level bandit framing. Treat each full completion $y \sim \pi_\theta(\cdot\mid x)$ as a single action; the reward $r_\phi(x,y)$ arrives at the end:
| Bandit concept | RLHF |
|---|---|
| Context | Prompt $x$ |
| Action | Full completion $y$ |
| Reward | $r_\phi(x, y)$ |
Goal:
$$\theta^* \;=\; \arg\max_\theta\;\mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta}\!\bigl[\,r_\phi(x, y)\,\bigr].$$
Score-function gradient. Since $y$ is sampled, the gradient of an expectation is rewritten as an expectation we can estimate from samples — using $\nabla_\theta \pi_\theta = \pi_\theta \nabla_\theta \log \pi_\theta$ (the log-derivative trick):
$$\nabla_\theta\, \mathbb{E}_{y \sim \pi_\theta}\!\bigl[r_\phi(x,y)\bigr] \;=\; \mathbb{E}_{y \sim \pi_\theta}\!\bigl[\,\nabla_\theta \log \pi_\theta(y \mid x)\,\cdot\, r_\phi(x,y)\,\bigr]$$
This is REINFORCE: sample $y$, weight $\nabla \log \pi_\theta(y|x)$ by its reward. We'll add KL regularization next.
3c. The KL-regularized objective¶
Why naive policy gradient fails. Maximizing $\mathbb{E}[r_\phi]$ alone leads to reward hacking: $\pi_\theta$ drifts into regions where $r_\phi$ was never trained and exploits its errors.
Fix. Anchor the policy to the (trusted) reference $\pi_\text{ref}$ via a KL penalty:
$$\boxed{\;J(\pi_\theta) \;=\; \mathbb{E}_{x,\, y \sim \pi_\theta}\!\bigl[r_\phi(x,y)\bigr] \;-\; \beta\, D_\text{KL}\!\bigl(\pi_\theta(\cdot|x)\,\Vert\,\pi_\text{ref}(\cdot|x)\bigr)\;}$$
$\beta$ trades off optimization vs. drift: small $\beta \Rightarrow$ aggressive, risk of hacking; large $\beta \Rightarrow$ stays near $\pi_\text{ref}$, limited improvement.
Closed-form optimum (one-line Lagrangian):
$$\pi^*(y|x) \;=\; \frac{1}{Z(x)}\,\pi_\text{ref}(y|x)\,e^{r_\phi(x,y)/\beta}, \qquad Z(x) = \sum_{y'} \pi_\text{ref}(y'|x)\, e^{r_\phi(x,y')/\beta}.$$
Inverting and taking logs gives the reward as a policy log-ratio:
$$r_\phi(x,y) \;=\; \beta \log \frac{\pi^*(y|x)}{\pi_\text{ref}(y|x)} \;+\; \beta \log Z(x).$$
The constant $\beta \log Z(x)$ depends on $x$ but not on $y$ — DPO will exploit exactly this.
Why we still need iterative optimization. $Z(x)$ is a sum over $|\mathcal V|^T$ sequences — intractable. We can't represent $\pi^*$ with a finite neural net, so we optimize $J(\pi_\theta)$ in $\theta$-space. PPO is the standard tool.
3d. Sequence-level PPO¶
The KL term in $J(\pi_\theta)$ becomes a per-completion correction, giving the shaped reward (computed once at rollout time):
$$\tilde r(x,y) \;=\; r_\phi(x,y) \;-\; \beta \log \frac{\pi_{\theta_\text{old}}(y|x)}{\pi_\text{ref}(y|x)}.$$
Why not just REINFORCE? Generating completions is expensive. We want to take multiple gradient steps per batch of rollouts — but once $\theta$ updates, those rollouts are off-policy.
PPO's fix. Snapshot $\theta_\text{old}$, sample once, and correct via importance sampling. With sequence-level ratio
$$\rho(\theta) \;=\; \frac{\pi_\theta(y|x)}{\pi_{\theta_\text{old}}(y|x)},$$
the clipped surrogate is
$$\boxed{\;\mathcal L_\text{PPO}(\theta) \;=\; \mathbb{E}_{x,\, y \sim \pi_{\theta_\text{old}}}\!\Bigl[\,\min\!\bigl(\rho(\theta)\,A(x,y),\;\; \mathrm{clip}(\rho,\, 1{-}\epsilon,\, 1{+}\epsilon)\,A(x,y)\bigr)\Bigr]\;}$$
with sequence-level advantage built from a per-prompt mean baseline:
$$A(x,y) \;=\; \tilde r(x,y) - \bar R(x), \qquad \bar R(x) = \tfrac{1}{K}\sum_{k=1}^K \tilde r(x, y^{(k)}).$$
The min makes the surrogate pessimistic: clipping zeroes the gradient only when an unclipped step would push outside the trust region.
Training loop. Repeat: (1) snapshot $\theta_\text{old} \leftarrow \theta$; (2) sample $K$ completions per prompt from $\pi_{\theta_\text{old}}$; (3) compute $\tilde r$ and $A$; (4) take $K_\text{ppo}$ gradient steps on $\mathcal L_\text{PPO}$.
This is exactly the bandit-PPO setup of RLOO (Ahmadian et al., 2024) and GRPO (Shao et al., 2024).
3e. Do we need RL? The closed-form optimal policy of the KL-regularized objective¶
Recall the KL-regularized RL objective:
$$\pi^* \;=\; \arg\max_{\pi}\; \mathbb{E}_{x,\, y \sim \pi}\bigl[r(x,y)\bigr] \;-\; \beta\, D_\text{KL}\!\bigl(\pi(\cdot|x)\,\|\,\pi_\text{ref}(\cdot|x)\bigr).$$
This problem admits an analytical solution (e.g., Ziegler et al., 2019):
$$\boxed{\;\pi^*(y \mid x) \;=\; \frac{1}{Z(x)}\,\pi_\text{ref}(y \mid x)\; e^{r(x,y)/\beta}\;}$$
where $Z(x) = \sum_{y'} \pi_\text{ref}(y'|x)\, e^{r(x,y')/\beta}$ is the (intractable) partition function.
- Simple derivation. Take a functional derivative of the Lagrangian $\mathbb E_\pi[r] - \beta\,\mathbb E_\pi[\log \pi/\pi_\text{ref}] - \lambda(x)\bigl(\sum_y \pi(y|x)-1\bigr)$ with respect to $\pi(y|x)$ and set it to zero:
$$r(x,y) - \beta\,\bigl[\log \pi^*(y|x) - \log \pi_\text{ref}(y|x)\bigr] - \beta - \lambda(x) = 0.$$
Solve for $\pi^*$ and absorb constants into $Z(x)$ via the simplex constraint.
The optimal policy is just the reference reweighted by the exponentiated reward; $1/\beta$ controls the steepness.
Inverting: reward as a log-ratio. Taking logs and rearranging:
$$\boxed{\;r(x, y) \;=\; \beta \log \frac{\pi^*(y \mid x)}{\pi_\text{ref}(y \mid x)} \;+\; \beta \log Z(x)\;}$$
The reward is the log-ratio of optimal-to-reference policies, plus a context-dependent constant $\beta \log Z(x)$ that depends on $x$ but not on $y$.
Important caveat. When we replace $\pi^*$ with a neural network $\pi_\theta$ on the right-hand side, the equality becomes an implicit modeling assumption: that the parameterization is rich enough to express the optimal closed form. DPO may be brittle because of this assumption.
Trick to eliminate $Z(x)$: Because $\beta \log Z(x)$ does not depend on $y$, it will cancel when we take reward differences across two responses to the same prompt, eliminating the partition function entirely.
4a. RL Not Needed: Direct Preference Optimization (DPO)¶
The insight (Rafailov et al., Direct Preference Optimization, NeurIPS 2023):
Substitute $r(x, y) = \beta \log \frac{\pi(y|x)}{\pi_\text{ref}(y|x)} + \beta \log Z(x)$ into the Bradley-Terry preference probability:
$$P_\theta(y^+ \succ y^- \mid x) = \sigma\!\bigl(r(x,y^+) - r(x,y^-)\bigr)$$
The $\beta \log Z(x)$ terms cancel because they are the same for $y^+$ and $y^-$:
$$P_\theta(y^+ \succ y^- \mid x) = \sigma\!\left(\beta \log \frac{\pi_\theta(y^+|x)}{\pi_\text{ref}(y^+|x)} - \beta \log \frac{\pi_\theta(y^-|x)}{\pi_\text{ref}(y^-|x)}\right)$$
This gives the DPO loss — a supervised loss directly on the policy parameters:
$$\boxed{\mathcal{L}_\text{DPO}(\theta) = -\mathbb{E}_{(x,\, y^+,\, y^-)}\!\left[\log \sigma\!\left(\beta \underbrace{\log \frac{\pi_\theta(y^+|x)}{\pi_\text{ref}(y^+|x)}}_{\text{log-ratio, chosen}} - \beta \underbrace{\log \frac{\pi_\theta(y^-|x)}{\pi_\text{ref}(y^-|x)}}_{\text{log-ratio, rejected}}\right)\right]}$$
No RL algorithm. No explicit reward model. Train the LLM directly on pairwise preference data.
4. What DPO does to the policy¶
The loss does two things simultaneously:
- Increases $\log \pi_\theta(y^+|x)$ relative to $\pi_\text{ref}(y^+|x)$: promotes chosen responses
- Decreases $\log \pi_\theta(y^-|x)$ relative to $\pi_\text{ref}(y^-|x)$: demotes rejected responses
The objective is to widen the log-ratio gap between $y^+$ and $y^-$:
$$\underbrace{\log \frac{\pi_\theta(y^+|x)}{\pi_\text{ref}(y^+|x)}}_{\text{should be large and positive}} \gg \underbrace{\log \frac{\pi_\theta(y^-|x)}{\pi_\text{ref}(y^-|x)}}_{\text{should be small or negative}}$$
DPO is "direct" in parameterization, not in statistical assumptions.
The loss looks simple, but it implicitly assumes:
- The Bradley-Terry noise model
- The KL-regularized control formulation with $\pi_\text{ref}$ as anchor
- The assumption that preference dataset $\mathcal{D}$ comes from same distribution as test data in deployment
| RLHF (PPO) | DPO | |
|---|---|---|
| Reward model | Explicit $r_\phi$ | Implicit: $r = \beta \log(\pi / \pi_\text{ref})$ |
| RL algorithm | PPO | None |
| Preference noise model | Bradley-Terry | Bradley-Terry |
| KL regularization | Explicit penalty | Implicit via $\pi_\text{ref}$ |
| Compute cost | High (two models + RL loop) | Low (supervised pass) |
Key insight. DPO and RLHF share all statistical assumptions. The difference is computational, not principled.
5. When DPO breaks: perfect separation¶
The logistic-loss pathology in perfectly separable data.
Suppose the physician always prefers $y^+$ over $y^-$ — i.e., preferences are (near-)deterministic. The DPO loss $-\log \sigma(z)$, viewed as a function of the log-ratio gap $z$, has no finite minimizer: it strictly decreases as $z \to +\infty$.
This is the same pathology as logistic regression on linearly separable data: the maximum-likelihood estimate is at infinity, and gradient descent will keep marching toward it.
Consequently, training drives
$$z \;\equiv\; \beta \log \frac{\pi_\theta(y^+|x)}{\pi_\text{ref}(y^+|x)} \;-\; \beta \log \frac{\pi_\theta(y^-|x)}{\pi_\text{ref}(y^-|x)} \;\longrightarrow\; +\infty.$$
This forces
$$\frac{\pi_\theta(y^-|x)}{\pi_\text{ref}(y^-|x)} \;\longrightarrow\; 0 \quad \text{for every rejected response } y^-,$$
so the policy collapses probability on all rejected outputs — including ones that were only slightly worse, not catastrophically bad. The implicit KL constraint via $\pi_\text{ref}$ does not save us, because the ratio $\pi_\theta/\pi_\text{ref}$ itself is being driven to zero.
Why this isn't a hyperparameter problem. Under deterministic labels, the unique minimizer of the logistic loss lies at infinity. No choice of $\beta$, learning rate, or weight decay can change that — the issue is structural in the loss function.
Three ways out:
- Label smoothing / preference noise. Replace $\sigma(z)$ with a smoothed target so the loss has a finite minimum (e.g., conservative DPO, cDPO). Equivalent to assuming the physician occasionally flips $y^+ \leftrightarrow y^-$.
- Parameter regularization (e.g., L2 on $\theta - \theta_\text{ref}$). Standard for small models, impractical for full-scale LLMs because $\theta$ has billions of parameters and there is no natural scale for the regularizer.
- Change the loss function: Identity Preference Optimization (next slide), which replaces the unbounded logistic with a bounded squared-error target.
5b. IPO: A robustness-oriented fix¶
Identity Preference Optimization (Azar et al., AISTATS 2024) modifies the objective to prevent unbounded growth of the log-ratio gap.
Rather than minimizing log-loss on the preference probability, IPO regresses the log-ratio difference to a fixed finite target $\tfrac{1}{2\beta}$:
$$\mathcal{L}_\text{IPO}(\theta) = \mathbb{E}_{(x, y^+, y^-)}\!\left[\!\left(\log \frac{\pi_\theta(y^+|x)}{\pi_\text{ref}(y^+|x)} - \log \frac{\pi_\theta(y^-|x)}{\pi_\text{ref}(y^-|x)} - \frac{1}{2\beta}\right)^{\!2}\right]$$
The choice $\tfrac{1}{2\beta}$ is not arbitrary: it is the unique stationary point of the population-level regression objective derived in the original paper, so optimizing this loss is consistent with the underlying preference model.
Key structural difference:
| DPO | IPO | |
|---|---|---|
| Loss as ratio gap $\to \infty$ | $\to 0$ (decreases without bound) | $\to \infty$ (quadratic blow-up) |
| Loss minimum at | Gap $= \infty$ | Gap $= 1/(2\beta)$ |
| KL collapse under deterministic labels | Can occur | Prevented |
| Surrogate / link | Logistic (unbounded) | Squared error (bounded) |
Statistical interpretation. IPO does not change the noise model; it changes the estimation principle — from maximum-likelihood under Bradley-Terry (DPO) to direct regression on the log-ratio gap (IPO). The squared-error target has a unique finite minimizer regardless of how one-sided the preference data is, so the implicit KL constraint via $\pi_\text{ref}$ remains well-defined.
Practical rule. Near-deterministic preferences ⇒ IPO, or DPO with strong $\beta$ and label smoothing. Genuinely noisy preferences ⇒ vanilla DPO is fine.
Note on squared error. The usual concern about L2 loss for binary classification (penalty when well classified) does not apply here. IPO is performing regression on a real-valued target (the log-ratio gap), not classification.
6. Beyond pairwise data: KTO¶
The practical problem. Collecting paired data $(x, y^+, y^-)$ requires the annotator to see both responses simultaneously. This is expensive and restricts annotation workflows. Can we learn from unpaired binary feedback?
KTO (Kahneman-Tversky Optimization; Ethayarajh et al., ICML 2024) works with individual labeled outputs:
$$\mathcal{D} = \{(x_i,\, y_i,\, \ell_i)\}_{i=1}^N, \quad \ell_i \in \{0\text{ (undesirable)},\, 1\text{ (desirable)}\}$$
The modeling choice: prospect theory.
KTO derives its objective from Kahneman and Tversky's value function. Humans are loss-averse: a given quality drop hurts more than an equal quality gain helps. The utility function is asymmetric:
$$u(x, y) = \begin{cases} \lambda_D \cdot \sigma\!\bigl(r(x,y) - z_\text{ref}\bigr) & \text{if } \ell = 1 \\ \lambda_U \cdot \sigma\!\bigl(z_\text{ref} - r(x,y)\bigr) & \text{if } \ell = 0 \end{cases}$$
where $z_\text{ref} = \beta\, D_\text{KL}(\pi_\theta \| \pi_\text{ref})$ is a reference point (the KL-adjusted value of the current policy) and $\lambda_U > \lambda_D$ encodes loss aversion.
- Important: Without the KL term $z_\text{ref}$, the optimal solution for undesirable samples would be to collapse probability mass to zero everywhere on bad outputs, trivially satisfying $u = 0$ without learning anything.
- The KL regularizer is not optional here; it is structurally necessary for the rejected-sample objective to be non-degenerate.
7. Taxonomy¶
All four main methods differ along a single statistical axis: what observation model for human feedback is assumed, and how that model is fit.
| Method | Feedback type | Noise model | Estimation principle | KL constraint |
|---|---|---|---|---|
| RLHF (PPO) | Paired $(y^+, y^-)$ | Bradley-Terry | MLE on rewards, then PPO on policy | Explicit penalty at policy stage |
| DPO | Paired $(y^+, y^-)$ | Bradley-Terry | MLE on policy log-ratios | Implicit via $\pi_\text{ref}$ in the loss |
| IPO | Paired $(y^+, y^-)$ | Bradley-Terry | Regression of log-ratio gap to $1/(2\beta)$ | Bounded by construction of the L2 target |
| KTO | Unpaired $(y, \ell)$ | Prospect theory (asymmetric utility) | Loss-aversion-weighted utility | Structurally necessary in the rejected-sample term |
The "RL vs non-RL" framing is misleading. DPO uses no RL algorithm at all, yet it makes identical statistical assumptions to RLHF.
The more important design choices are not "do we use RL?", instead they are: (i) what noise model do we assume for feedback, and (ii) how do we constrain the policy from drifting away from $\pi_\text{ref}$, which is a meta-learning question.
8. Relation to contrastive learning¶
Contrastive learning is the generic machinery for learning from relative comparisons: increase the score for positive examples and decrease the score for negatives.
Preference optimization is a special case of contrastive learning:
| Contrastive representation learning | Preference optimization | |
|---|---|---|
| Positive | Matching views / similar items | Preferred output $y^+$ |
| Negative | Mismatched items | Rejected output $y^-$ |
| Score | Embedding dot product $z_x^\top z_y$ | Log-prob ratio $\log\!\bigl(\pi_\theta(y\mid x)/\pi_\text{ref}(y\mid x)\bigr)$ |
| Loss | InfoNCE | BT logistic (DPO) / squared error (IPO) |
| Regularizer | Uniformity on hypersphere / temperature | KL divergence to $\pi_\text{ref}$ |
An exact correspondence. With a single negative, the loss reduces to
$$-\log \frac{e^{s_+}}{e^{s_+} + e^{s_-}} \;=\; -\log \sigma(s_+ - s_-),$$
which is literally the DPO loss when $s_+ - s_- = \beta\bigl[\log\frac{\pi_\theta(y^+)}{\pi_\text{ref}(y^+)} - \log\frac{\pi_\theta(y^-)}{\pi_\text{ref}(y^-)}\bigr]$. DPO is this loss applied to policy log-ratios.
- Pair informativeness. When $y^+$ and $y^-$ are nearly indistinguishable in quality, $P(y^+ \succ y^- \mid x) \approx 0.5$ and the gradient of the BT likelihood is near zero.
- Weakly separated pairs carry almost no information about the latent reward boundary. This is the statistical basis for prioritizing pairs with large quality gaps during data curation, and for hard-negative mining in contrastive representation learning.
9. All methods are variants of the same thing¶
All four methods are, in the end, estimating a relative energy function $\Delta E$ from comparison data:
$$\Delta E(x, y^+, y^-) \;=\; r(x, y^+) \;-\; r(x, y^-)$$
| Method | What is estimated | How | |---|---|---| | RLHF | $\Delta E$, explicitly via a separate model $r_\phi$ | MLE on $r_\phi$, then PPO on $\pi_\theta$ | | DPO | $\Delta E$, implicitly via $\beta \cdot \bigl[\log\!\tfrac{\pi_\theta(y^+)}{\pi_\text{ref}(y^+)} - \log\!\tfrac{\pi_\theta(y^-)}{\pi_\text{ref}(y^-)}\bigr]$ | Single MLE step; $Z(x)$ cancels | | IPO | $\Delta E$ via the same log-ratio gap as DPO | L2 regression to $1/(2\beta)$ | | KTO | The implicit reward $r_\theta(x,y) = \beta \log\!\tfrac{\pi_\theta(y|x)}{\pi_\text{ref}(y|x)}$, per sample, anchored against the running KL reference $z_\text{ref}$ | Loss-aversion-weighted utility on individual labels |
- Contrastive learning is the generic machinery for fitting such relative energies from comparison data.
- RL (policy gradient, PPO) is the machinery for converting an estimated energy function into a sequential-decision policy.
One-sentence synthesis.
Preference optimization learns a relative energy function from noisy pairwise comparisons; the choice of noise model and link function distinguishes RLHF, DPO, IPO, and KTO; and the KL regularizer (explicit or implicit) prevents the optimizer from exploiting model misspecification in all four cases.
Summary¶
The feedback is pairwise comparison. The noise model for these comparisons (not whether RL is used) is what determines the method.
RLHF is a two-stage pipeline. Stage 1 fits a Bradley-Terry reward model from pairwise data. Stage 2 optimizes a KL-regularized policy with PPO. Reward hacking makes the KL constraint mandatory, not optional.
The KL-regularized objective has a closed form: $\pi^*(y|x) \propto \pi_\text{ref}(y|x)\, e^{r(x,y)/\beta}$. Inverting this identity expresses the reward as a log-ratio of policies plus a context-dependent constant $\beta \log Z(x)$.
DPO collapses both stages by substituting the closed form into the Bradley-Terry likelihood; the partition function $Z(x)$ cancels in the reward difference.
- The result is a supervised loss on policy log-ratios. DPO and RLHF share all statistical assumptions; the difference is computational.
The DPO pathology is structural. The logistic link is unbounded: under near-deterministic preferences, the log-ratio gap grows without bound and the implicit regularization through $\pi_\text{ref}$ becomes ineffective. This is not a hyperparameter issue. Two ways out: (a) add label noise (cDPO), or (b) change the loss — IPO uses a bounded squared-error target with a finite minimizer at $1/(2\beta)$.
KTO relaxes the paired-data requirement using prospect theory (Kahneman–Tversky asymmetric utility). The KL term is structurally necessary in KTO's rejected-sample objective: without it, the loss has a trivial degenerate minimum at $\pi_\theta \to 0$ on rejected outputs.
All four methods are contrastive estimation of a relative energy function. The choice of noise model, link/loss, and regularizer fully characterizes the family.
References¶
- Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345. https://doi.org/10.1093/biomet/39.3-4.324
- Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems (NeurIPS), 30. https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html
- Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. https://doi.org/10.48550/arXiv.1707.06347
- Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., ... & Ziegler, D. M. (2019). Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. https://doi.org/10.48550/arXiv.1909.08593
- Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., ... & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 35, 27730–27744. https://proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html
- Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems (NeurIPS), 36, 53728–53741. https://proceedings.neurips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html
- Gheshlaghi Azar, M., Daniel Guo, Z., Piot, B., Munos, R., Rowland, M., Valko, M., & Calandriello, D. (2024). A general theoretical paradigm to understand learning from human preferences. Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS), 238, 4447–4455. https://proceedings.mlr.press/v238/gheshlaghi-azar24a.html
- Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024). KTO: Model alignment as prospect theoretic optimization. Proceedings of the 41st International Conference on Machine Learning (ICML). https://arxiv.org/abs/2402.01306