Machine Learning Fundamentals · free preview

Weight Decay and Regularization

Weight decay adds a penalty on the magnitude of the parameters to the training objective, which trades a worse fit on the training data for a smaller generalization gap.

1. The penalized objective

The penalized objective

2. Why a worse train fit is the point

Why a worse train fit is the point

3. Early stopping

Early stopping

4. What the penalty term does to the gradient

The weight-decayed objective adds λ2∥w∥2\frac{\lambda}{2}\|\mathbf{w}\|^2 to the usual loss. Which statement describes the effect on the parameter update?

5. The ridge objective

The ridge objective adds the term ____ to the unregularized training loss, where λ\lambda controls the strength of the penalty.

Practice this interactively

Free account · instant grading · spaced review that schedules itself.

Start this course for free