Machine Learning Fundamentals · free preview
Momentum on an Ill-Conditioned Loss
When the loss curves far more sharply in one direction than another, gradient descent cannot take a step that is safe in every direction, and momentum averages past gradients to keep moving along the flat one.
1. A loss with a ravine
A loss with a ravine
2. Averaging the past gradients
Averaging the past gradients
3. What momentum does and does not fix
What momentum does and does not fix
4. Why momentum helps on a ravine
Gradient descent on zigzags, bouncing across the steep direction while advancing slowly along the flat one. Which statement describes what momentum changes?
5. The velocity update
With a momentum coefficient , a previous velocity of and a current gradient of , the new velocity equals ____.
Practice this interactively
Free account · instant grading · spaced review that schedules itself.
Start this course for free