Content
Summary
An activation function is a nonlinear function applied to a neuron's weighted sum of inputs. Without it, stacked layers collapse into a single linear map. Common choices are sigmoid, tanh, and ReLU, each with different output ranges and gradient behavior.
Intuition
Think of a neuron as computing a score z from its inputs, then deciding how strongly to 'fire'. A linear neuron simply passes the score along, so a deep network of them can only draw straight lines or flat planes. An activation function bends the score: sigmoid squashes it to between 0 and 1 like a probability, tanh squashes it between -1 and 1, and ReLU switches off negative scores and passes positive ones unchanged. These bends let layers combine into curved, complex decision boundaries, much like a medical test threshold turning a continuous biomarker level into a graded response.
Key equations
A neuron computes a weighted sum z, then applies activation f to get output a.
Sigmoid squashes values to (0,1); its gradient is at most 0.25.
Tanh outputs values in (-1,1), is zero-centered, with maximum gradient 1.
Passes positive inputs unchanged and outputs zero for negative inputs; gradient is 1 for z>0 and 0 for z<0.
Derivation
- Take two layers with no activation: a1 = W1 x + b1, then a2 = W2 a1 + b2.
- Substitute: a2 = W2(W1 x + b1) + b2 = (W2 W1) x + (W2 b1 + b2).
- This is W' x + b', a single linear layer, so depth adds no expressive power.
- Insert a nonlinear f between layers: a2 = W2 f(W1 x + b1) + b2. This cannot be reduced to one linear map in general.
- Derivative of sigmoid: with s = 1/(1+e^{-z}), ds/dz = e^{-z}/(1+e^{-z})^2 = s(1-s).
- Since s(1-s) is at most 0.25 (at z=0) and approaches 0 for large |z|, gradients shrink across many sigmoid layers, causing vanishing gradients.
- ReLU has gradient 1 for positive z, so gradients do not shrink there, which is why it is the default in deep networks.
Worked example
A neuron has weights w = (2, -1), bias b = 0.5, and input x = (1, 3). Compute the output using ReLU, sigmoid, and tanh.
- Compute z = 2*1 + (-1)*3 + 0.5 = 2 - 3 + 0.5 = -0.5.
- ReLU: max(0, -0.5) = 0.
- Sigmoid: 1/(1+e^{0.5}) = 1/(1+1.6487) ≈ 0.378.
- Tanh: (e^{-0.5} - e^{0.5})/(e^{-0.5}+e^{0.5}) = (0.6065-1.6487)/(0.6065+1.6487) ≈ -0.462.
Answer z = -0.5; ReLU = 0, sigmoid ≈ 0.378, tanh ≈ -0.462.
Common mistakes
- Believing deeper linear networks are more powerful than a single linear layer.
- Using sigmoid in many hidden layers and being surprised by vanishing gradients.
- Forgetting that ReLU gradient is zero for negative inputs, which can cause 'dead neurons'.
- Confusing the output range: sigmoid is (0,1), tanh is (-1,1).
- Applying a hidden-layer activation to the output layer when the task needs a different one, such as softmax for classes.