AI, machine learning and deep learning · Introductory

Gradient descent

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Gradient descent is an iterative method for minimizing a function by repeatedly stepping in the direction opposite to the gradient. It is the core algorithm used to train machine learning models by reducing a loss. The learning rate controls how big each step is.

Intuition

Imagine standing on a foggy hillside and wanting to reach the valley. You cannot see the whole landscape, but you can feel which way the ground slopes under your feet. If you always step downhill, steeply where the slope is steep and gently where it is shallow, you will eventually reach a low point. The gradient is the mathematical slope that points uphill, so we move in its opposite direction. Step size matters: too small and progress is painfully slow, too large and you overshoot the valley and may bounce back and forth or even diverge. In machine learning the hill is the loss surface, and the position is the set of model parameters.

Key equations

θt+1=θt−η∇L(θt)
(1)

Update rule: new parameters equal old parameters minus learning rate times gradient of the loss.

∇L=(∂L∂θ1,…,∂L∂θn)
(2)

The gradient collects the partial derivatives and points in the direction of steepest increase.

L(θ−ηg)≈L(θ)−η‖g‖2
(3)

For small eta, a step against the gradient g decreases the loss.

Derivation

  1. Goal: find parameters θ that minimize the loss L(θ).
  2. Near the current point, a first-order Taylor expansion gives L(θ+d)≈L(θ)+g⋅d, where g is the gradient.
  3. To make L decrease most for a small step of fixed length, choose d to make g·d as negative as possible.
  4. That happens when d points opposite to g, so d = -eta g with eta > 0.
  5. Substituting gives L(θ−ηg)≈L(θ)−η|g|2, which is a decrease whenever g is not zero.
  6. Repeat the update until the gradient is near zero or the loss stops improving.

Worked example

Problem

Minimize L(w) = (w - 3)^2 starting at w = 0 with learning rate 0.1. Do three steps.

  1. The gradient is dL/dw = 2(w - 3).
  2. Step 1: gradient at w=0 is -6, so w = 0 - 0.1(-6) = 0.6.
  3. Step 2: gradient at 0.6 is -4.8, so w = 0.6 + 0.48 = 1.08.
  4. Step 3: gradient at 1.08 is -3.84, so w = 1.08 + 0.384 = 1.464.
  5. Each step shrinks the distance to 3 by a factor of 0.8, so w approaches 3.

Answer After three steps w = 1.464, moving steadily toward the minimum at w = 3.

Common mistakes

  • Adding the gradient instead of subtracting it, which climbs the hill (gradient ascent).
  • Choosing a learning rate that is too large, causing oscillation or divergence.
  • Assuming gradient descent always finds the global minimum; it can stop at local minima or saddle points on non-convex losses.
  • Forgetting to update all parameters simultaneously using the gradient at the old point.
  • Stopping because the loss fluctuates slightly, when using noisy minibatch gradients.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(62 lines)
import sys
from manim import *
from blocks import *
import numpy as np


def f(x):
    """Loss function: a simple parabola with minimum at x=2."""
    return (x - 2.0) ** 2 + 0.5


def grad(x):
    """Derivative of f."""
    return 2.0 * (x - 2.0)


def gradient_descent(x0, lr, steps):
    """Run gradient descent and record every x visited."""
    xs = [x0]
    for _ in range(steps):
        xs.append(xs[-1] - lr * grad(xs[-1]))
    return xs


TRACE = gradient_descent(-1.0, 0.3, 8)


class Main(Scene):
    def construct(self):
        # Title
        TitleCard(self, "Gradient descent", "Walk downhill on the loss")
        # Plot of the loss
        plot = CurvePlot(f, x_range=(-2, 6), y_range=(0, 10, 2), x_label="w",
                         y_label="loss", x_length=8, y_length=4.2)
        plot.shift(DOWN * 0.1)
        self.play(Create(plot.axes), FadeIn(plot.labels), Create(plot.curve))
        StepCaption(self, "Start at w = -1 and follow the negative slope")
        # Animate recorded states
        dot = plot.dot_at(TRACE[0])
        self.play(FadeIn(dot))
        info = Text("", font_size=24).to_corner(UR, buff=0.5)
        for k in range(1, len(TRACE)):
            x = TRACE[k]
            new_pos = plot.axes.c2p(x, f(x))
            lab = Text(f"step {k}: w = {x:.2f}", font_size=24, color=C_ACCENT)
            lab.to_corner(UR, buff=0.5)
            trail = Dot(plot.axes.c2p(TRACE[k - 1], f(TRACE[k - 1])),
                        color=C_DIM, radius=0.05)
            self.add(trail)
            self.play(dot.animate.move_to(new_pos), Transform(info, lab),
                      run_time=0.7)
        StepCaption(self, "Steps shrink as the slope flattens near the minimum")
        self.wait(1)


if __name__ == "__main__" and "--selftest" in sys.argv:
    assert abs(TRACE[0] - (-1.0)) < 1e-12
    assert abs(TRACE[-1] - 2.0) < 0.1
    losses = [f(x) for x in TRACE]
    assert all(a > b for a, b in zip(losses, losses[1:]))
    assert abs(grad(2.0)) < 1e-12
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.