Feedforward Networks of Neural Units

Feedforward Networks of Neural Units

Hidden layers learn representations

Forward passes in NumPy

A unit has weights $\mathbf{w} = [1, 0.5, -1]$, input $\mathbf{x} = [2, 3, 3]$ and bias $b = -1$. Compute its ReLU activation, $\max(\mathbf{w} \cdot \mathbf{x} + b, 0)$, as a decimal.

Which pairing of activation property and consequence is correct?

Suppose every activation in a 5-layer network were replaced by the identity, leaving pure linear layers. What function class can the resulting network compute?

In a two-unit ReLU hidden layer, the inputs $[0, 1]$ and $[1, 0]$ both produce the hidden vector $[0, 0]$, while $[0, 0]$ produces $[0, 1]$ and $[1, 1]$ produces $[1, 0]$. In this hidden space, are the XOR-positive points and the XOR-negative points linearly separable? Answer yes or no.

A classifier has input dimension $n_0 = 6$, $n_1 = 3$ hidden units with ReLU, and $n_2 = 4$ output classes with softmax. How many rows does the first weight matrix $\mathbf{{W}}$ have? Answer with the number.