Backpropagation and Training

Backpropagation and Training

Backprop for a two-layer network

Backward differentiation in NumPy

On the graph $d = 4a$, $e = d + c$, $L = b \cdot e$ with values $a = 3$, $b = 2$, $c = 4$, compute $\frac{\partial L}{\partial a}$ by the chain rule. Answer as a decimal.

Compute $\frac{d\sigma}{dz}$ at $z = -0.5$, using $\sigma(z)(1 - \sigma(z))$, and round to three decimals.

The output error is $\delta^{[2]} = 0.4$, the output weight feeding hidden unit 1 is $w = -0.5$, and that hidden unit received pre-activation $z = 0.5$. Compute $\delta^{[1]} = w \cdot \delta^{[2]} \cdot \mathbb{1}[z > 0]$ as a decimal.

For a sigmoid output unit with cross-entropy loss, what is the error signal $\delta = \frac{{\partial L}}{{\partial z}}$ at the output logit, and why is it so clean?

A hand-written backward pass produces analytic gradients. Which procedure validates them, and what outcome should it produce?