AI, machine learning and deep learning · Intermediate

Shallow neural networks: general case

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Shallow neural networks with a single hidden layer map multi-dimensional inputs to outputs by combining linear transformations with non-linear activation functions, resulting in a continuous piecewise linear function. This architecture creates numerous convex linear regions in the input space, with the count growing exponentially with input dimensions and hidden units, allowing for the approximation of complex continuous functions.

Intuition

Imagine a shallow neural network as a geometric sculptor working in high-dimensional space. Each hidden unit acts like a knife that slices through the input space along a hyperplane, defined by the weights and bias. Because of non-linear activation functions like ReLU, these slices create 'kinks' or joints where the function's behavior changes. The output is a continuous surface made of many flat, convex polygonal pieces stitched together. As you add more input dimensions or hidden units, the number of these flat regions explodes exponentially, allowing the network to approximate incredibly complex shapes. However, while shallow networks can theoretically approximate any continuous function, deep networks often achieve similar results with far fewer parameters, making them more efficient for structured data like images.

Key equations

hd=a[θd0+∑i=1Diθdixi]
(1)

Computation of the d-th hidden unit, where a is the activation function, θd0 is the bias, θdi are weights, and x_i are inputs.

y=ϕ0+∑d=1Dϕdhd
(2)

Output computation as a linear combination of hidden units, where ϕ0 is the output bias and ϕd are output weights.

h=a[θ0+θx]
(3)

Matrix notation for the hidden layer, where θ0 is the bias vector and θ is the weight matrix.

Derivation

  1. Start with a multi-dimensional input vector x∈ℝDi.
  2. Compute each hidden unit hd by applying an activation function a[⋅] to a linear combination of inputs and a bias: hd=a[θd0+∑i=1Diθdixi].
  3. Recognize that each hidden unit defines a hyperplane in the input space, creating a 'kink' where the unit activates or deactivates.
  4. Combine the hidden units linearly to produce the final output: y=ϕ0+∑d=1Dϕdhd.
  5. Conclude that the resulting function is continuous and piecewise linear, composed of convex regions whose count grows exponentially with Di and D.

Worked example

Problem

Consider a shallow network with input dimension Di=100 and hidden units D=500. Estimate the potential number of linear regions and explain the implication for representational capacity.

  1. Identify the parameters: Di=100 and D=500.
  2. Recall that the number of linear regions grows exponentially with input dimensions and hidden units.
  3. Refer to the specific example provided in the source: with these dimensions, the network can create over 10107 regions.
  4. Interpret this number: it demonstrates immense representational capacity, allowing the network to model highly complex continuous functions.

Answer The network can create over 10107 linear regions, demonstrating immense representational capacity despite being 'small' by modern standards.

Common mistakes

  • Assuming that more linear regions always imply better performance; the structure of these regions and symmetries matter.
  • Believing that deep networks are always easier to train; training becomes difficult again with many layers.
  • Thinking that shallow networks cannot approximate complex functions; they can, but may require exponentially more units than deep networks.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.