AI, machine learning and deep learning · Intermediate

Adam optimizer path

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Adam is a gradient-based optimizer that keeps running averages of the gradient (momentum) and of the squared gradient (scale), then takes steps normalized per parameter. This produces smooth, well-scaled paths across ravines and flat regions where plain gradient descent zigzags or crawls.

Intuition

Imagine a ball rolling down a bumpy, stretched valley. Plain gradient descent reacts only to the slope right under it, so it bounces between steep walls and moves slowly along the gentle floor. Adam gives the ball two abilities. First, it remembers recent slopes, so back-and-forth components cancel while the consistent downhill direction builds up. Second, it tracks how large the slope has typically been in each coordinate and divides by that, so steep directions get smaller steps and shallow directions get relatively larger ones. The result is a path that heads more directly toward the minimum, with step size roughly set by the learning rate rather than by gradient magnitude.

Key equations

mt=β1mt−1+(1−β1)gt
(1)

Exponential moving average of gradients (first moment, momentum).

vt=β2vt−1+(1−β2)gt2
(2)

Exponential moving average of squared gradients (second moment), elementwise.

m^t=mt1−β1t,v^t=vt1−β2t
(3)

Bias correction, since m and v start at zero.

θt=θt−1−αm^tv^t+ϵ
(4)

Parameter update with per-parameter normalized step.

Derivation

  1. Start with m_0=0 and v_0=0. Unrolling the average gives m_t=(1-β1)Σ β1^(t-i) g_i.
  2. If gradients have roughly constant expectation E[g], then E[m_t]=E[g](1-β1^t), so m_t is biased toward zero early on.
  3. Dividing by (1-β1^t) removes this bias; the same argument applies to v_t with β2.
  4. The ratio m̂/√v̂ has magnitude about 1 when gradients are consistent in sign, so each step is about α in size per coordinate.
  5. When gradients fluctuate in sign, m̂ is small relative to √v̂, so the step shrinks, damping oscillation.
  6. ε (about 1e-8) prevents division by zero when v̂ is tiny.

Worked example

Problem

One parameter, θ_0=1, gradient g_1=0.5 at step 1. Use α=0.1, β1=0.9, β2=0.999, ε≈0. Compute θ_1.

  1. m_1=0.9·0+0.1·0.5=0.05.
  2. v_1=0.999·0+0.001·0.25=0.00025.
  3. m̂_1=0.05/(1-0.9)=0.5.
  4. v̂_1=0.00025/(1-0.999)=0.25, so √v̂_1=0.5.
  5. Step=0.1·0.5/0.5=0.1.
  6. θ_1=1-0.1=0.9.

Answer θ_1 = 0.9; the first step has size α regardless of gradient scale.

Common mistakes

  • Forgetting bias correction, which makes early steps too small.
  • Thinking Adam's step is proportional to the gradient size; it is roughly normalized to α.
  • Assuming Adam always beats SGD; it can generalize worse and may need learning-rate decay.
  • Applying L2 regularization through the gradient instead of using decoupled weight decay (AdamW).
  • Reading v as a variance; it is an uncentered second moment.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(75 lines)
import sys
import numpy as np
from manim import *
from blocks import *

# Loss surface: elongated bowl f(x,y) = 0.5*(x^2 + 10*y^2)
def grad(p):
    return np.array([p[0], 10.0 * p[1]])

def loss(p):
    return 0.5 * (p[0] ** 2 + 10.0 * p[1] ** 2)

def adam(start, lr=0.3, b1=0.9, b2=0.999, eps=1e-8, steps=25):
    """Run Adam with fixed settings, recording every position."""
    p = np.array(start, dtype=float)
    m = np.zeros(2)
    v = np.zeros(2)
    path = [p.copy()]
    for t in range(1, steps + 1):
        g = grad(p)
        m = b1 * m + (1 - b1) * g
        v = b2 * v + (1 - b2) * g * g
        mh = m / (1 - b1 ** t)
        vh = v / (1 - b2 ** t)
        p = p - lr * mh / (np.sqrt(vh) + eps)
        path.append(p.copy())
    return path

PATH = adam([-3.0, 1.2])


class Main(Scene):
    def construct(self):
        self.camera.background_color = "#111418"
        TitleCard(self, "Adam optimizer path", "Momentum + per-parameter scaling")

        # Axes with contour ellipses of the loss
        axes = Axes(x_range=[-4, 4, 2], y_range=[-2, 2, 1], x_length=8, y_length=4,
                    axis_config={"color": C_DIM, "include_tip": False})
        axes.shift(DOWN * 0.2)
        contours = VGroup()
        for c in [0.5, 1.5, 3, 5, 7]:
            a = np.sqrt(2 * c)
            b = np.sqrt(2 * c / 10)
            contours.add(Ellipse(width=2 * a * axes.x_axis.unit_size,
                                 height=2 * b * axes.y_axis.unit_size,
                                 color=C_MAIN, stroke_width=2,
                                 stroke_opacity=0.6).move_to(axes.c2p(0, 0)))
        self.play(Create(axes), Create(contours), run_time=1.5)

        # Draw the recorded path step by step
        pts = [axes.c2p(p[0], p[1]) for p in PATH]
        dot = Dot(pts[0], color=C_ACCENT, radius=0.1)
        self.play(FadeIn(dot))
        info = Text("loss = %.2f" % loss(PATH[0]), font_size=26, color=WHITE)
        info.to_corner(UR, buff=0.5).shift(DOWN * 0.6)
        self.add(info)
        for i in range(1, len(pts)):
            seg = Line(pts[i - 1], pts[i], color=C_ACCENT, stroke_width=3)
            new = Text("loss = %.2f" % loss(PATH[i]), font_size=26, color=WHITE)
            new.move_to(info)
            self.play(Create(seg), dot.animate.move_to(pts[i]), Transform(info, new),
                      run_time=0.4, rate_func=linear)
        self.wait(0.5)
        StepCaption(self, "Adam damps the steep y direction and still reaches the minimum.")
        self.wait(1.5)


if __name__ == "__main__" and "--selftest" in sys.argv:
    assert len(PATH) == 26
    assert np.allclose(PATH[0], [-3.0, 1.2])
    # first Adam step has size ~lr in each coordinate
    assert np.allclose(np.abs(PATH[1] - PATH[0]), 0.3, atol=1e-3)
    assert loss(PATH[-1]) < loss(PATH[0]) * 0.2
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.