Gradient Descent on the Cross-Entropy Loss

Gradient Descent on the Cross-Entropy Loss

Following the gradient downhill

Training a tiny sentiment classifier

A classifier outputs $\hat{y} = 0.9$ and the gold label is $y = 0$. Compute the cross-entropy loss $L_{\text{CE}}(\hat{y}, y)$ and round to three decimals.

During training, a document has feature value $x_j = 5$, the model outputs $\hat{y} = 0.7$, and the gold label is $y = 0$. Compute $\frac{\partial L}{\partial w_j} = (\hat{y} - y)\,x_j$ and round to three decimals.

A weight starts at $w_j = 1$, the document's feature is $x_j = 4$, the model outputs $\hat{y} = 0.3$, the gold label is $y = 1$, and the learning rate is $\eta = 0.1$. After the SGD update $w_j \leftarrow w_j - \eta (\hat{y} - y) x_j$, what is the new $w_j$? Round to three decimals.

Both documents have the same feature value $x_j = 3$ and gold label $y = 0$. Document A gets $\hat{y} = 0.1$ and document B gets $\hat{y} = 0.9$. Which weight update has the larger magnitude, and why?

Which statement about training logistic regression with gradient descent is correct?