Machine Learning Fundamentals · free preview

Cross-Entropy and Its Gradient

Softmax cross-entropy is the negative log-likelihood of the correct class, and its gradient with respect to the scores has the same shape as linear regression's: the feature vector times a residual.

1. From probability to loss

From probability to loss

2. The simplified form

The simplified form

3. The gradient

The gradient

4. What the loss charges

A three-class model assigns probability y^2=0.7\hat{y}_2 = 0.7 to the true class, whose one-hot label is (0,1,0)(0, 1, 0). The per-example loss is −∑jyjlog⁡y^j-\sum_j y_j \log \hat{y}_j. The value of that loss is closest to which number?

5. The gradient at the true class

A model's softmax output for the true class is y^2=0.7\hat{y}_2 = 0.7 with label y2=1y_2 = 1. The gradient component ∂l/∂o2=y^2−y2\partial l / \partial o_2 = \hat{y}_2 - y_2 equals ____.

Practice this interactively

Free account · instant grading · spaced review that schedules itself.

Start this course for free