Content
Summary
The perceptron is a single artificial neuron that classifies inputs into two classes using a weighted sum and a threshold. The perceptron learning rule nudges the weights whenever a training example is misclassified, moving the decision boundary toward the correct side. If the classes are linearly separable, this procedure is guaranteed to converge.
Intuition
Imagine a line separating two groups of points. The perceptron draws a line using its weights. When it labels a point wrongly, it tilts and shifts the line a little so that the point is more likely to fall on the correct side next time. If a positive point is called negative, we add a fraction of that point's input vector to the weights, making the weighted sum bigger for that input. If a negative point is called positive, we subtract. Correct points cause no change. Repeating this over the data, the line wiggles until no point is misclassified, provided a separating line exists.
Key equations
Prediction: threshold of the weighted sum of inputs plus bias.
Weight update: change is proportional to the error and the input.
Bias update uses the same error with a constant input of 1.
Derivation
- Define the net input z = wĀ·x + b and predict 1 if z ā„ 0, else 0.
- Error is e = y ā Å·, which is 0 if correct, +1 if a true 1 was predicted 0, and ā1 if a true 0 was predicted 1.
- We want to change z in the direction that reduces the error: increase z when e = +1, decrease when e = ā1.
- Updating w by Ī·eĀ·x changes z by Ī·e(xĀ·x) = Ī·eāxā², which has the sign of e, so z moves the right way.
- Apply the same logic to b, treating it as a weight on a constant input of 1.
- When e = 0 nothing changes, so updates occur only on mistakes.
Worked example
Start with w = (0, 0), b = 0, Ī· = 1. Sample A: x = (1, 2), y = 1. Sample B: x = (2, 1), y = 0. Present A, then B, then A again. Give the final weights and bias and whether A is then correct.
- Sample A: z = 0, so Å· = 1 (since z ā„ 0). Correct, e = 0, no change.
- Sample B: z = 0, so Å· = 1 but y = 0, e = ā1.
- Update w = (0,0) + (ā1)(2,1) = (ā2, ā1); b = 0 ā 1 = ā1.
- Sample A again: z = (ā2)(1) + (ā1)(2) ā 1 = ā5, so Å· = 0 but y = 1, e = +1.
- Update w = (ā2,ā1) + (1,2) = (ā1, 1); b = ā1 + 1 = 0.
- Check A: z = ā1 + 2 + 0 = 1 ā„ 0, so Å· = 1, correct. Check B: z = ā2 + 1 = ā1, Å· = 0, correct.
Answer w = (ā1, 1), b = 0; both samples are now classified correctly.
Common mistakes
- Updating weights on correctly classified points.
- Forgetting to update the bias along with the weights.
- Using the wrong sign of the error, which pushes the boundary away from the point.
- Expecting convergence on data that is not linearly separable, such as XOR.
- Confusing the perceptron rule with gradient descent on a smooth loss; the step output is not differentiable.