Content
Summary
A recurrent neural network applies the same cell repeatedly across a sequence, carrying a hidden state forward. Unrolling in time draws each step as a copy of the network with shared weights, which makes forward computation and backpropagation through time easy to trace.
Intuition
Think of reading a sentence word by word while keeping a running mental summary. At each word you update the summary using the new word and the old summary. The RNN does exactly this: the hidden state is the summary, and one set of weights defines how to update it. Unrolling simply lays out each time step side by side, so a loop becomes a long feed-forward chain whose layers share parameters. Gradients flow backward along this chain, and because they are multiplied by the recurrent Jacobian at every step, they can shrink (vanish) or grow (explode) with sequence length.
Key equations
Hidden state update using the current input and previous hidden state, with the same weights at every step.
Output at time t computed from the hidden state.
Total loss is the sum of per-step losses.
Gradient path from step t back to step k is a product of Jacobians, source of vanishing or exploding gradients.
Derivation
- Start with the loop: h_t = f(h_{t-1}, x_t; θ) with h_0 given, usually zeros.
- Unroll: h_1 = f(h_0,x_1), h_2 = f(h_1,x_2),..., h_T = f(h_{T-1},x_T). Every f uses the same θ.
- Compute outputs y_t from each h_t and sum the losses L_t.
- Because θ appears at every step, the gradient is a sum over steps: dL/dW_hh = sum_t sum_{k<=t} (dL_t/dh_t)(dh_t/dh_k)(∂h_k/∂W_hh).
- The factor dh_t/dh_k expands by the chain rule into a product of (t-k) Jacobians, each containing W_hh and the tanh derivative.
- If the largest singular value of these factors is below 1, the product decays exponentially; if above 1, it can blow up. Clipping gradients or using gated cells (LSTM, GRU) mitigates this.
Worked example
A scalar RNN has h_t = tanh(0.5 h_{t-1} + x_t), h_0 = 0, inputs x_1 = 1, x_2 = 0. Compute h_1, h_2, and dh_2/dh_1.
- h_1 = tanh(0.5·0 + 1) = tanh(1) ≈ 0.7616.
- h_2 = tanh(0.5·0.7616 + 0) = tanh(0.3808) ≈ 0.3632.
- Derivative: dh_2/dh_1 = (1 - h_2^2)·0.5.
- 1 - 0.3632^2 = 1 - 0.1319 = 0.8681.
- dh_2/dh_1 = 0.8681 × 0.5 ≈ 0.434.
Answer h_1 ≈ 0.762, h_2 ≈ 0.363, dh_2/dh_1 ≈ 0.434 (each step back multiplies the gradient by roughly this factor, so it shrinks).
Common mistakes
- Thinking each unrolled step has its own weights; they are shared, so gradients are summed across steps.
- Forgetting to initialize or carry the hidden state h_0 between steps.
- Confusing the unrolled diagram with a very deep different network; it is one cell reused.
- Assuming RNNs remember arbitrarily long context; vanishing gradients limit this.
- Applying gradient clipping to fix vanishing gradients; clipping only addresses exploding ones.