Content
Summary
A diffusion model gradually corrupts data with Gaussian noise over steps according to a noising schedule that sets how much signal survives at each step. The schedule, defined by or , determines the signal-to-noise ratio trajectory and strongly affects training and sample quality.
Intuition
Imagine slowly turning a photo into static. The forward process adds a small amount of noise at each step, and because Gaussians compose, you can jump directly to any step in closed form. The schedule is the pace of this destruction. A linear beta schedule destroys information quickly in the later steps, so many steps are wasted on nearly pure noise. A cosine schedule keeps changing smoothly, so signal-to-noise ratio falls more evenly and the model learns useful denoising at all noise levels. Thinking in terms of log signal-to-noise ratio makes schedules comparable: a schedule is just a monotone curve of SNR against time, and the sampler walks that curve backwards.
Key equations
One forward noising step with variance beta_t.
Cumulative signal retention after t steps.
Closed-form sampling of x_t directly from x_0.
Signal-to-noise ratio at step t; schedules are curves of this quantity.
Cosine schedule (Nichol and Dhariwal), with small offset s.
Derivation
- Define a single step: , with .
- Substitute .
- The two noise terms are independent Gaussians with variances alpha_t(1-alpha_{t-1}) and 1-alpha_t; their sum has variance 1 - alpha_t alpha_{t-1}.
- So .
- Repeat by induction to get .
- Variance is preserved when has unit variance: .
- Hence the schedule is fully captured by , or equivalently .
Worked example
Linear schedule with T=1000, beta from 1e-4 to 0.02. Approximate alpha-bar at t=500 and the SNR.
- Beta at step s is about 1e-4 + (0.02-1e-4)(s/1000) ≈ 1e-4 + 0.0199 s/1000.
- for small (more precisely ).
- Sum of beta over 500 steps ≈ 500·1e-4 + 0.0199·(500²/2)/1000 = 0.05 + 2.4875 = 2.5375.
- Second-order correction: sum beta²/2 is about 0.0199²/(2·10⁶)·(500³/3)=0.0199²·41.7/2·... ≈ 0.008 (small), so log alpha-bar ≈ -2.546.
- -bar .
- SNR = 0.078/(1-0.078) ≈ 0.085.
Answer alpha-bar_500 ≈ 0.08 and SNR ≈ 0.085 (log SNR ≈ -2.5): by the midpoint the signal is already mostly destroyed.
Common mistakes
- Confusing (per-step noise) with (cumulative signal retention).
- Summing betas instead of multiplying for ; the sum is only a small- approximation.
- Assuming a schedule tuned for 64x64 images works unchanged at higher resolution; higher resolution needs more noise to reach the same SNR.
- Believing the final step reaches exactly zero SNR; with a non-zero terminal SNR, train-sampling mismatch leaks signal (fixed by zero-terminal-SNR rescaling).
- Using too many steps at near-pure-noise levels, wasting model capacity.