Content
Summary
An artificial neuron takes several numerical inputs, multiplies each by a weight, adds a bias, and passes the result through a nonlinear activation function. A forward pass is simply this computation carried out from inputs to output. Geometrically, a neuron splits input space with a line (or plane) and the activation shapes the response.
Intuition
Think of a neuron as a tiny voting machine. Each input casts a vote, and its weight says how much that input matters and in which direction. The bias is a built-in tendency to fire or stay quiet. The votes are summed into one number, z. Geometrically, z is a signed distance-like measure from a line in input space: points on one side give positive z, points on the other give negative z. The activation function, such as a sigmoid, then squashes z into a useful range like 0 to 1, so the neuron reports how confidently a point lies on the positive side. Stacking neurons into layers and feeding outputs forward gives a network, and the forward pass is just repeating this recipe layer by layer.
Key equations
Weighted sum of inputs plus bias (pre-activation).
Sigmoid activation squashes z into the range 0 to 1.
The decision boundary: a line in the (x1, x2) plane where z = 0.
Forward pass for a whole layer using a weight matrix.
Derivation
- Start with inputs x1 and x2 arranged as a point in the plane.
- Give each input a weight: the weight vector w = (w1, w2) points in a direction in this plane.
- The dot product w·x measures how far x extends along the direction of w, scaled by its length.
- Adding the bias b shifts this value, so the zero level moves away from the origin.
- Setting z = 0 gives the line w1 x1 + w2 x2 + b = 0; w is perpendicular to this line.
- Points with z > 0 lie on the side w points toward; z < 0 on the other side.
- Applying sigmoid turns z into a = σ(z); a = 0.5 exactly on the line and approaches 1 or 0 far from it.
- For many neurons, stack weight vectors as rows of W to compute all z values at once, then repeat for the next layer.
Worked example
A neuron has weights w = (2, -1), bias b = -1, and sigmoid activation. Compute its output for input x = (3, 2). Which side of the boundary is the point on?
- Compute the weighted sum: 2·3 + (-1)·2 = 6 - 2 = 4.
- Add the bias: z = 4 - 1 = 3.
- Apply sigmoid: a = 1/(1+e^{-3}).
- e^{-3} ≈ 0.0498, so a ≈ 1/1.0498 ≈ 0.953.
- Since z = 3 > 0, the point lies on the positive side of the line 2x1 - x2 - 1 = 0.
Answer z = 3, output a ≈ 0.95, on the positive side of the boundary.
Common mistakes
- Forgetting the bias, which forces the boundary through the origin.
- Thinking the weights are the inputs or that they change during a forward pass; they are fixed while computing.
- Confusing z (pre-activation) with a (post-activation).
- Believing a single neuron can separate any data; it can only draw one straight boundary.
- Omitting the nonlinearity: stacked linear layers collapse into a single linear map.