Natural Language Processing · 免费预览

基于指令与偏好的后训练

指令微调、使用LoRA的参数高效微调、偏好数据与「Bradley-Terry」模型、奖励模型、KL正则化对齐，以及DPO。

1. Post-Training with Instructions and Preferences

Post-Training with Instructions and Preferences

2. Learning from preferences

Learning from preferences

3. A toy reward model in NumPy

A toy reward model in NumPy

4. Bradley-Terry preference probability

Two responses have reward scores $z_w = 1.5$ and $z_l = 0$ . Compute $P(o_w \succ o_l) = \sigma(z_w - z_l)$ , rounding to three decimals.

5. Reward model loss on one pair

For one preference pair the reward model scores the winner $2$ above the loser. Compute the loss $-\log \sigma(2)$ , rounding to three decimals.

6. LoRA parameter count

A frozen weight matrix is $2048 \times 2048$ , and LoRA adapts it with matrices of shapes $2048 \times 8$ and $8 \times 2048$ . How many trainable parameters does LoRA add? Answer with the number.

7. Why the KL penalty

In preference-based alignment, the optimization maximizes reward minus $\beta$ times the KL divergence to a reference policy. What goes wrong with reward alone?

8. What DPO eliminates

Direct Preference Optimization optimizes the same KL-regularized objective as reward-model RL. What does it skip, and why is that possible?

进入交互式练习

免费注册 · 即时批改 · 自动安排间隔复习。

免费开始这门课