Content
Summary
Gradient descent is an iterative method for minimizing a function by repeatedly stepping in the direction opposite to the gradient. It is the core algorithm used to train machine learning models by reducing a loss. The learning rate controls how big each step is.
Intuition
Imagine standing on a foggy hillside and wanting to reach the valley. You cannot see the whole landscape, but you can feel which way the ground slopes under your feet. If you always step downhill, steeply where the slope is steep and gently where it is shallow, you will eventually reach a low point. The gradient is the mathematical slope that points uphill, so we move in its opposite direction. Step size matters: too small and progress is painfully slow, too large and you overshoot the valley and may bounce back and forth or even diverge. In machine learning the hill is the loss surface, and the position is the set of model parameters.
Key equations
Update rule: new parameters equal old parameters minus learning rate times gradient of the loss.
The gradient collects the partial derivatives and points in the direction of steepest increase.
For small eta, a step against the gradient g decreases the loss.
Derivation
- Goal: find parameters that minimize the loss .
- Near the current point, a first-order Taylor expansion gives , where is the gradient.
- To make L decrease most for a small step of fixed length, choose d to make g·d as negative as possible.
- That happens when d points opposite to g, so d = -eta g with eta > 0.
- Substituting gives , which is a decrease whenever is not zero.
- Repeat the update until the gradient is near zero or the loss stops improving.
Worked example
Minimize L(w) = (w - 3)^2 starting at w = 0 with learning rate 0.1. Do three steps.
- The gradient is dL/dw = 2(w - 3).
- Step 1: gradient at w=0 is -6, so w = 0 - 0.1(-6) = 0.6.
- Step 2: gradient at 0.6 is -4.8, so w = 0.6 + 0.48 = 1.08.
- Step 3: gradient at 1.08 is -3.84, so w = 1.08 + 0.384 = 1.464.
- Each step shrinks the distance to 3 by a factor of 0.8, so w approaches 3.
Answer After three steps w = 1.464, moving steadily toward the minimum at w = 3.
Common mistakes
- Adding the gradient instead of subtracting it, which climbs the hill (gradient ascent).
- Choosing a learning rate that is too large, causing oscillation or divergence.
- Assuming gradient descent always finds the global minimum; it can stop at local minima or saddle points on non-convex losses.
- Forgetting to update all parameters simultaneously using the gradient at the old point.
- Stopping because the loss fluctuates slightly, when using noisy minibatch gradients.