Content
Summary
Shallow neural networks with a single hidden layer map multi-dimensional inputs to outputs by combining linear transformations with non-linear activation functions, resulting in a continuous piecewise linear function. This architecture creates numerous convex linear regions in the input space, with the count growing exponentially with input dimensions and hidden units, allowing for the approximation of complex continuous functions.
Intuition
Imagine a shallow neural network as a geometric sculptor working in high-dimensional space. Each hidden unit acts like a knife that slices through the input space along a hyperplane, defined by the weights and bias. Because of non-linear activation functions like ReLU, these slices create 'kinks' or joints where the function's behavior changes. The output is a continuous surface made of many flat, convex polygonal pieces stitched together. As you add more input dimensions or hidden units, the number of these flat regions explodes exponentially, allowing the network to approximate incredibly complex shapes. However, while shallow networks can theoretically approximate any continuous function, deep networks often achieve similar results with far fewer parameters, making them more efficient for structured data like images.
Key equations
Computation of the d-th hidden unit, where a is the activation function, is the bias, are weights, and x_i are inputs.
Output computation as a linear combination of hidden units, where is the output bias and are output weights.
Matrix notation for the hidden layer, where is the bias vector and is the weight matrix.
Derivation
- Start with a multi-dimensional input vector .
- Compute each hidden unit by applying an activation function to a linear combination of inputs and a bias: .
- Recognize that each hidden unit defines a hyperplane in the input space, creating a 'kink' where the unit activates or deactivates.
- Combine the hidden units linearly to produce the final output: .
- Conclude that the resulting function is continuous and piecewise linear, composed of convex regions whose count grows exponentially with and .
Worked example
Consider a shallow network with input dimension and hidden units . Estimate the potential number of linear regions and explain the implication for representational capacity.
- Identify the parameters: and .
- Recall that the number of linear regions grows exponentially with input dimensions and hidden units.
- Refer to the specific example provided in the source: with these dimensions, the network can create over regions.
- Interpret this number: it demonstrates immense representational capacity, allowing the network to model highly complex continuous functions.
Answer The network can create over linear regions, demonstrating immense representational capacity despite being 'small' by modern standards.
Common mistakes
- Assuming that more linear regions always imply better performance; the structure of these regions and symmetries matter.
- Believing that deep networks are always easier to train; training becomes difficult again with many layers.
- Thinking that shallow networks cannot approximate complex functions; they can, but may require exponentially more units than deep networks.