Content
Summary
Adam is a gradient-based optimizer that keeps running averages of the gradient (momentum) and of the squared gradient (scale), then takes steps normalized per parameter. This produces smooth, well-scaled paths across ravines and flat regions where plain gradient descent zigzags or crawls.
Intuition
Imagine a ball rolling down a bumpy, stretched valley. Plain gradient descent reacts only to the slope right under it, so it bounces between steep walls and moves slowly along the gentle floor. Adam gives the ball two abilities. First, it remembers recent slopes, so back-and-forth components cancel while the consistent downhill direction builds up. Second, it tracks how large the slope has typically been in each coordinate and divides by that, so steep directions get smaller steps and shallow directions get relatively larger ones. The result is a path that heads more directly toward the minimum, with step size roughly set by the learning rate rather than by gradient magnitude.
Key equations
Exponential moving average of gradients (first moment, momentum).
Exponential moving average of squared gradients (second moment), elementwise.
Bias correction, since m and v start at zero.
Parameter update with per-parameter normalized step.
Derivation
- Start with m_0=0 and v_0=0. Unrolling the average gives m_t=(1-β1)Σ β1^(t-i) g_i.
- If gradients have roughly constant expectation E[g], then E[m_t]=E[g](1-β1^t), so m_t is biased toward zero early on.
- Dividing by (1-β1^t) removes this bias; the same argument applies to v_t with β2.
- The ratio m̂/√v̂ has magnitude about 1 when gradients are consistent in sign, so each step is about α in size per coordinate.
- When gradients fluctuate in sign, m̂ is small relative to √v̂, so the step shrinks, damping oscillation.
- ε (about 1e-8) prevents division by zero when v̂ is tiny.
Worked example
One parameter, θ_0=1, gradient g_1=0.5 at step 1. Use α=0.1, β1=0.9, β2=0.999, ε≈0. Compute θ_1.
- m_1=0.9·0+0.1·0.5=0.05.
- v_1=0.999·0+0.001·0.25=0.00025.
- m̂_1=0.05/(1-0.9)=0.5.
- v̂_1=0.00025/(1-0.999)=0.25, so √v̂_1=0.5.
- Step=0.1·0.5/0.5=0.1.
- θ_1=1-0.1=0.9.
Answer θ_1 = 0.9; the first step has size α regardless of gradient scale.
Common mistakes
- Forgetting bias correction, which makes early steps too small.
- Thinking Adam's step is proportional to the gradient size; it is roughly normalized to α.
- Assuming Adam always beats SGD; it can generalize worse and may need learning-rate decay.
- Applying L2 regularization through the gradient instead of using decoupled weight decay (AdamW).
- Reading v as a variance; it is an uncentered second moment.