Machine Learning Fundamentals · free preview

Momentum on an Ill-Conditioned Loss

When the loss curves far more sharply in one direction than another, gradient descent cannot take a step that is safe in every direction, and momentum averages past gradients to keep moving along the flat one.

1. A loss with a ravine

A loss with a ravine

2. Averaging the past gradients

Averaging the past gradients

3. What momentum does and does not fix

What momentum does and does not fix

4. Why momentum helps on a ravine

Gradient descent on f(x)=0.1x12+2x22f(\mathbf{x}) = 0.1x_1^2 + 2x_2^2 zigzags, bouncing across the steep direction while advancing slowly along the flat one. Which statement describes what momentum changes?

5. The velocity update

With a momentum coefficient β=0.9\beta = 0.9, a previous velocity of 00 and a current gradient of 22, the new velocity vt=βvt−1+∇Lv_t = \beta v_{t-1} + \nabla \mathcal{L} equals ____.

Practice this interactively

Free account · instant grading · spaced review that schedules itself.

Start this course for free