AI, machine learning and deep learning · Intermediate

RNN unrolled in time

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

A recurrent neural network applies the same cell repeatedly across a sequence, carrying a hidden state forward. Unrolling in time draws each step as a copy of the network with shared weights, which makes forward computation and backpropagation through time easy to trace.

Intuition

Think of reading a sentence word by word while keeping a running mental summary. At each word you update the summary using the new word and the old summary. The RNN does exactly this: the hidden state is the summary, and one set of weights defines how to update it. Unrolling simply lays out each time step side by side, so a loop becomes a long feed-forward chain whose layers share parameters. Gradients flow backward along this chain, and because they are multiplied by the recurrent Jacobian at every step, they can shrink (vanish) or grow (explode) with sequence length.

Key equations

ht=tanh⁡(Wxhxt+Whhht−1+bh)
(1)

Hidden state update using the current input and previous hidden state, with the same weights at every step.

yt=Whyht+by
(2)

Output at time t computed from the hidden state.

L=∑t=1TLt(yt,y^t)
(3)

Total loss is the sum of per-step losses.

∂ht∂hk=∏j=k+1tdiag(1−hj2)Whh
(4)

Gradient path from step t back to step k is a product of Jacobians, source of vanishing or exploding gradients.

Derivation

  1. Start with the loop: h_t = f(h_{t-1}, x_t; θ) with h_0 given, usually zeros.
  2. Unroll: h_1 = f(h_0,x_1), h_2 = f(h_1,x_2),..., h_T = f(h_{T-1},x_T). Every f uses the same θ.
  3. Compute outputs y_t from each h_t and sum the losses L_t.
  4. Because θ appears at every step, the gradient is a sum over steps: dL/dW_hh = sum_t sum_{k<=t} (dL_t/dh_t)(dh_t/dh_k)(∂h_k/∂W_hh).
  5. The factor dh_t/dh_k expands by the chain rule into a product of (t-k) Jacobians, each containing W_hh and the tanh derivative.
  6. If the largest singular value of these factors is below 1, the product decays exponentially; if above 1, it can blow up. Clipping gradients or using gated cells (LSTM, GRU) mitigates this.

Worked example

Problem

A scalar RNN has h_t = tanh(0.5 h_{t-1} + x_t), h_0 = 0, inputs x_1 = 1, x_2 = 0. Compute h_1, h_2, and dh_2/dh_1.

  1. h_1 = tanh(0.5·0 + 1) = tanh(1) ≈ 0.7616.
  2. h_2 = tanh(0.5·0.7616 + 0) = tanh(0.3808) ≈ 0.3632.
  3. Derivative: dh_2/dh_1 = (1 - h_2^2)·0.5.
  4. 1 - 0.3632^2 = 1 - 0.1319 = 0.8681.
  5. dh_2/dh_1 = 0.8681 × 0.5 ≈ 0.434.

Answer h_1 ≈ 0.762, h_2 ≈ 0.363, dh_2/dh_1 ≈ 0.434 (each step back multiplies the gradient by roughly this factor, so it shrinks).

Common mistakes

  • Thinking each unrolled step has its own weights; they are shared, so gradients are summed across steps.
  • Forgetting to initialize or carry the hidden state h_0 between steps.
  • Confusing the unrolled diagram with a very deep different network; it is one cell reused.
  • Assuming RNNs remember arbitrarily long context; vanishing gradients limit this.
  • Applying gradient clipping to fix vanishing gradients; clipping only addresses exploding ones.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(180 lines)
import sys
import numpy as np
from manim import *
from blocks import *

# --- RNN Simulation Logic ---

def rnn_step(h_prev, x, W_xh, W_hh, b_h, W_hy, b_y):
    """
    Standard RNN step:
    h_t = tanh(W_xh * x_t + W_hh * h_{t-1} + b_h)
    y_t = W_hy * h_t + b_y
    """
    h_t = np.tanh(W_xh * x + W_hh * h_prev + b_h)
    y_t = W_hy * h_t + b_y
    return h_t, y_t

def simulate_rnn(sequence, W_xh, W_hh, b_h, W_hy, b_y):
    """
    Unroll the RNN over the sequence and record states.
    Returns a list of dicts: {t, x, h, y}
    """
    states = []
    h = 0.0  # Initial hidden state
    for t, x in enumerate(sequence):
        h, y = rnn_step(h, x, W_xh, W_hh, b_h, W_hy, b_y)
        states.append({
            "t": t,
            "x": x,
            "h": h,
            "y": y
        })
    return states

# Fixed weights for deterministic trace
W_xh = 0.5
W_hh = 0.3
b_h = 0.1
W_hy = 1.5
b_y = 0.0

# Input sequence
INPUT_SEQ = [1.0, -0.5, 0.8, -1.2]

# Compute trace
TRACE = simulate_rnn(INPUT_SEQ, W_xh, W_hh, b_h, W_hy, b_y)

class Main(Scene):
    def construct(self):
        self.camera.background_color = "#101418"
        
        # Title
        TitleCard(self, "RNN Unrolling", "Recurrent Neural Network in time")
        
        # Layout setup
        # We will draw the unrolled chain: x1->h1->y1, x2->h2->y2...
        # To keep it readable, we'll use a GridView-like manual layout or just VGroups
        
        # Define colors
        C_X = "#58C4DD"   # Input
        C_H = "#FFD166"   # Hidden
        C_Y = "#83E377"   # Output
        C_W = "#FF6B6B"   # Weights/Connections
        
        # Create nodes for each time step
        # We'll place them in a row
        node_radius = 0.25
        step_width = 2.0
        start_x = -((len(TRACE) - 1) * step_width) / 2
        
        # Store mobjects for animation
        time_steps = []
        
        # Build the visual components for each step
        for i, state in enumerate(TRACE):
            t = state["t"]
            x_val = state["x"]
            h_val = state["h"]
            y_val = state["y"]
            
            x_pos = start_x + t * step_width
            
            # X Node
            x_node = Circle(radius=node_radius, fill_color=C_X, fill_opacity=0.8, stroke_color=WHITE, stroke_width=2)
            x_node.move_to([x_pos, 0, 0])
            x_lab = Text(f"{x_val:.1f}", font_size=20, color=BLACK)
            x_lab.move_to(x_node)
            
            # H Node
            h_node = Circle(radius=node_radius, fill_color=C_H, fill_opacity=0.8, stroke_color=WHITE, stroke_width=2)
            h_node.move_to([x_pos, -1.5, 0])
            h_lab = Text(f"{h_val:.2f}", font_size=18, color=BLACK)
            h_lab.move_to(h_node)
            
            # Y Node
            y_node = Circle(radius=node_radius, fill_color=C_Y, fill_opacity=0.8, stroke_color=WHITE, stroke_width=2)
            y_node.move_to([x_pos, -3.0, 0])
            y_lab = Text(f"{y_val:.2f}", font_size=18, color=BLACK)
            y_lab.move_to(y_node)
            
            # Connections
            # x -> h
            conn_xh = Line(x_node.get_bottom(), h_node.get_top(), color=C_W, stroke_width=2)
            # h -> y
            conn_hy = Line(h_node.get_bottom(), y_node.get_top(), color=C_W, stroke_width=2)
            # h(t-1) -> h(t) (recurrent)
            conn_rec = None
            if t > 0:
                prev_h_node = time_steps[t-1]["h_node"]
                conn_rec = Line(prev_h_node.get_right(), h_node.get_left(), color=C_W, stroke_width=2)
            
            # Labels for weights (small text near connections)
            lab_xh = Text("W_xh", font_size=14, color=C_DIM).next_to(conn_xh, LEFT, buff=0.1)
            lab_hy = Text("W_hy", font_size=14, color=C_DIM).next_to(conn_hy, RIGHT, buff=0.1)
            lab_hh = None
            if conn_rec:
                lab_hh = Text("W_hh", font_size=14, color=C_DIM).next_to(conn_rec, UP, buff=0.1)

            time_steps.append({
                "t": t,
                "x_node": x_node, "x_lab": x_lab,
                "h_node": h_node, "h_lab": h_lab,
                "y_node": y_node, "y_lab": y_lab,
                "conn_xh": conn_xh, "conn_hy": conn_hy, "conn_rec": conn_rec,
                "lab_xh": lab_xh, "lab_hy": lab_hy, "lab_hh": lab_hh,
                "state": state
            })

        # Assemble initial scene (empty)
        all_mobs = VGroup()
        for ts in time_steps:
            all_mobs.add(ts["x_node"], ts["x_lab"], ts["h_node"], ts["h_lab"],
                         ts["y_node"], ts["y_lab"], ts["conn_xh"], ts["conn_hy"],
                         ts["lab_xh"], ts["lab_hy"])
            if ts["conn_rec"]:
                all_mobs.add(ts["conn_rec"], ts["lab_hh"])
        
        # Animate step by step
        for i, ts in enumerate(time_steps):
            # Highlight current time index
            StepCaption(self, f"Time step {i}: x={ts['state']['x']:.1f}")
            
            # Fade in X
            self.play(FadeIn(ts["x_node"]), FadeIn(ts["x_lab"]))
            self.play(Create(ts["conn_xh"]), FadeIn(ts["lab_xh"]))
            
            # Compute/Show H (visualize activation)
            self.play(ts["h_node"].animate.set_fill(C_H, opacity=0.8), FadeIn(ts["h_lab"]))
            
            # Recurrent connection if applicable
            if ts["conn_rec"]:
                self.play(Create(ts["conn_rec"]), FadeIn(ts["lab_hh"]))
            
            # Show Y
            self.play(Create(ts["conn_hy"]), FadeIn(ts["y_node"]), FadeIn(ts["y_lab"]), FadeIn(ts["lab_hy"]))
            
            # Brief pause to read values
            self.wait(0.5)

        StepCaption(self, "Hidden state h carries memory from previous steps")
        self.wait(1)

if __name__ == "__main__" and "--selftest" in sys.argv:
    # Verify trace correctness
    h_prev = 0.0
    for i, state in enumerate(TRACE):
        t = state["t"]
        x = state["x"]
        
        # Manual calculation
        h_calc = np.tanh(W_xh * x + W_hh * h_prev + b_h)
        y_calc = W_hy * h_calc + b_y
        
        assert abs(h_calc - state["h"]) < 1e-10, f"Step {t}: h mismatch"
        assert abs(y_calc - state["y"]) < 1e-10, f"Step {t}: y mismatch"
        
        h_prev = h_calc
        
    assert len(TRACE) == len(INPUT_SEQ)
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.