Machine Learning Fundamentals · free preview

Backpropagation

Backpropagation applies the chain rule along the reversed computational graph, computing one gradient at a time and reusing each result for every consumer, which is why a network's gradient costs about the same as its forward pass.

1. The chain rule, applied in reverse

The chain rule, applied in reverse

2. One reverse pass in detail

One reverse pass in detail

3. Forward and backward as one loop

Forward and backward as one loop

4. Which stored value the output-weight gradient needs

In the reverse pass through the output layer, the gradient ∂J/∂W(2)=(∂J/∂o) h⊤\partial J/\partial \mathbf{W}^{(2)} = (\partial J/\partial \mathbf{o})\,\mathbf{h}^\top is computed. The vector h\mathbf{h} comes from which phase?

5. The gradient through the softmax and cross-entropy

For a softmax classifier with cross-entropy loss, the gradient of the objective with respect to the output scores o\mathbf{o} equals ____.

Practice this interactively

Free account · instant grading · spaced review that schedules itself.

Start this course for free