OmniPulse | AI, tech and markets

Preference alignment adapts model behavior using judgments about preferred outputs. RLHF uses a reward-based training stage; DPO directly optimizes an objective from preference pairs.

About 2 min to read · Take extra time for practice · AI-assisted learning notes · Expanded October 17, 2026

First, the key terms.

Preference pair: Two outputs with a judgment about which is preferred.
Reward model: A model estimating a score from preference-related data.

The idea, explained

Instruction following and preference training address behavior after broad pretraining. In a common RLHF pipeline, people compare responses, a reward model learns from those comparisons and a policy is optimized against that reward. DPO offers a different optimization formulation using preference data. Both depend on whose preferences are collected, how comparisons are framed and what is evaluated. Neither makes a model universally safe or ensures agreement with every user’s values.

STEP 1

Inspect the judgments

Reviewers may prefer clarity, helpfulness, honesty or brevity, but those qualities can conflict. The instructions and population of reviewers shape the collected preference data. A preference is not automatically objective truth.

STEP 2

Separate method from evaluation

RLHF commonly uses a reward-model and policy-optimization stage. DPO uses a different objective based on preference pairs. Evaluate both for the task and affected users rather than treating either name as proof of alignment.

Follow a worked example.

Reviewers prefer a clear answer that admits uncertainty over a longer invented explanation. Training on these comparisons may encourage that behavior, but the trained model still needs tests for unfamiliar questions and conflicting instructions.

Walk it through, one step at a time

Reviewers choose between a concise unsupported answer and a longer accurate answer.
If instructions emphasize brevity without factual checks, the dataset can reward the wrong response.
Improve the judgments and evaluate independently. Optimizing an imperfect signal harder can amplify its shortcomings.

Where people get stuck

Optimizing a preference proxy can reward surface qualities while missing factual accuracy or minority needs.

Now try it yourself.

Pause before opening each answer. Explain your reasoning in your own words.

1. Check the core idea

Does better preference performance prove factual correctness?

Reveal the answer & reasoning

No. Preference is a training signal, not a truth oracle. Factuality, fairness and robustness require separate evidence and evaluation.

2. Apply what you learned

Why must factual evaluation remain separate from preference wins?