Machine Learning Fundamentals · free preview
Parameter Initialization
Parameters start at random rather than zero, because zero-initialized weights in a layer make every unit identical and the layer learns one feature, and the distribution's scale controls whether activations and gradients grow or vanish.
1. The failure of zero initialization
The failure of zero initialization
2. The default scaling
The default scaling
3. What goes wrong without the balance
What goes wrong without the balance
4. Why zero initialization fails
A hidden layer of 256 units has every weight initialized to zero and every bias to zero. Training begins with minibatch gradient descent. Which statement describes what happens?
5. The default scale
A weight tensor has input units. The common initialization draws weights from a normal distribution with standard deviation , which equals ____.
Practice this interactively
Free account · instant grading · spaced review that schedules itself.
Start this course for free