Machine Learning Fundamentals · free preview

Hidden Layers and Activation Functions

A single affine map cannot express a relationship whose effect reverses direction, and composing affine maps with a nonlinearity between them removes the limitation.

1. What linearity assumes

What linearity assumes

2. A hidden layer

A hidden layer

3. Two standard activations

Two standard activations

4. Why a hidden layer needs a nonlinearity

An MLP layer computes H=σ(XW(1)+b(1))\mathbf{H} = \sigma(\mathbf{X}\mathbf{W}^{(1)} + \mathbf{b}^{(1)}) and a second layer computes O=HW(2)+b(2)\mathbf{O} = \mathbf{H}\mathbf{W}^{(2)} + \mathbf{b}^{(2)}. The activation σ\sigma is replaced by the identity, so H=XW(1)+b(1)\mathbf{H} = \mathbf{X}\mathbf{W}^{(1)} + \mathbf{b}^{(1)}. What is the effect on the model?

5. The ReLU at three inputs

The ReLU is max⁡(z,0)\max(z, 0). Its value at z=−2z = -2 is ____.

Practice this interactively

Free account · instant grading · spaced review that schedules itself.

Start this course for free