Preference Optimization

Preference optimization trains a model to prefer outputs that humans, users, tests, or another evaluator rank higher. It includes reward-model-based reinforcement learning and direct objectives over preference pairs.

For pairwise preference data $(y_w,y_l)$, a common loss is:

$$\mathcal{L}=-\log \sigma (r_{\theta }(x,y_w)-r_{\theta }(x,y_l))$$

Here $y_w$ is the preferred output, $y_l$ is the less preferred output, and $r_{\theta }$ is a reward or preference score.

Why It Matters

• Aligns model behavior with goals that are hard to specify as a loss.
• Common in the post-training stage of large generative models.
• Quality depends heavily on the preference data and reward signal.

Direct Preference Objective

Direct Preference Optimization (DPO)-style objectives compare a policy to a reference model:

$${\mathcal{L}}_{\mathrm{DPO}}=-{\mathbb{E}}_{(x,y_w,y_l)}\left[\log \sigma \left(\beta \log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\mathrm{ref}}(y_w\mid x)}-\beta \log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\mathrm{ref}}(y_l\mid x)}\right)\right]$$

where ${\pi }_{\mathrm{ref}}$ anchors the optimized policy and $\beta$ controls how strongly the policy can move away from the reference.

Preference Data Contract

Field	Question
Prompt or task	What condition $x$ is the preference about?
Compared outputs	Are outputs final answers, trajectories, code diffs, molecules, poses, or plans?
Preference source	human, expert, model judge, verifier, benchmark, or rule?
Label type	pairwise, ranking, scalar score, binary accept/reject, or edit preference?
Distribution	generated by which policy or sampler?
Tie and ambiguity policy	are equal or low-confidence preferences kept?
Safety/public boundary	does the data contain private prompts, people, paths, or unpublished details?

Preference optimization is only as meaningful as the comparison protocol.

Evidence Boundary

Claim	Required Evidence
better human preference	held-out preference evaluation with annotator or judge protocol
better task performance	task metric, execution result, verifier, or benchmark score
safer behavior	adversarial or policy-specific safety evaluation
less reward hacking	examples where optimized score and external quality agree
stable post-training	KL/reference drift, regression checks, and capability retention

Do not equate higher preference-model score with true quality unless external evaluation supports it.

Checks

• Are preference labels consistent and free of annotator bias?
• Does optimization reward-hack the proxy instead of the true objective?
• How much does the policy drift from the reference model?
• Is the preference signal collected from realistic tasks or artificial comparisons?
• Does the evaluation use fresh prompts not used for preference collection?
• Are win rates paired, confidence-bounded, and judged with a stated protocol?

Related

• Fine-tuning
• Reward modeling
• Policy gradient
• Reinforcement learning
• Supervised learning
• Autoregressive model
• Paired comparison
• Agent evaluation