Natural Language Processing · free preview

Post-Training with Instructions and Preferences

Instruction tuning, parameter-efficient fine-tuning with LoRA, preference data and the Bradley-Terry model, reward models, KL-regularized alignment, and DPO.

1. Post-Training with Instructions and Preferences

2. Learning from preferences

Learning from preferences

3. A toy reward model in NumPy

A toy reward model in NumPy

4. Bradley-Terry preference probability

Two responses have reward scores $z_w = 1.5$ and $z_l = 0$ . Compute $P(o_w \succ o_l) = \sigma(z_w - z_l)$ , rounding to three decimals.

5. Reward model loss on one pair

For one preference pair the reward model scores the winner $2$ above the loser. Compute the loss $-\log \sigma(2)$ , rounding to three decimals.

6. LoRA parameter count

A frozen weight matrix is $2048 \times 2048$ , and LoRA adapts it with matrices of shapes $2048 \times 8$ and $8 \times 2048$ . How many trainable parameters does LoRA add? Answer with the number.

7. Why the KL penalty

In preference-based alignment, the optimization maximizes reward minus $\beta$ times the KL divergence to a reference policy. What goes wrong with reward alone?

8. What DPO eliminates

Direct Preference Optimization optimizes the same KL-regularized objective as reward-model RL. What does it skip, and why is that possible?

Practice this interactively

Free account · instant grading · spaced review that schedules itself.

Start this course for free