Content
Summary
An autoencoder compresses an input x into a low-dimensional latent vector z through an encoder, then reconstructs x from z with a decoder. Training minimizes reconstruction error, so the latent space learns a compact, task-relevant coordinate system of the data manifold. Its geometry (smoothness, dimensionality, regularization) determines whether interpolation and generation are meaningful.
Intuition
Real data such as images or gene expression profiles do not fill their high-dimensional space; they lie near a low-dimensional curved surface, the manifold. The bottleneck forces the network to discard directions that carry no information for reconstruction and keep the few that do. Think of z as map coordinates: the encoder projects a point in a huge space onto a map, and the decoder lifts it back. A plain autoencoder only guarantees that points from the training set reconstruct well, so the space between them may be 'holes' that decode to nonsense. A variational autoencoder adds a KL penalty that pulls codes toward a Gaussian prior, filling holes and making the space smooth at some cost to reconstruction sharpness. A linear autoencoder with squared error recovers the same subspace as PCA.
Key equations
Encoder f maps input to latent code; decoder g maps code back to a reconstruction.
Reconstruction loss minimized over encoder and decoder parameters.
VAE evidence lower bound: reconstruction term minus a regularizer shaping the latent space.
Reparameterization trick allowing gradients through sampling.
Derivation
- Assume data x in R^D lies near a d-dimensional manifold with d much smaller than D.
- Choose an encoder f mapping R^D to R^d and a decoder g mapping R^d to R^D.
- Define the loss as mean squared error between x and g(f(x)); gradient descent updates both networks by backpropagation through the bottleneck.
- For linear f=Wx and g=Vz, the optimum has V W as the projection onto the top d principal components, so the latent space spans the PCA subspace (up to an invertible transform).
- With nonlinear networks the latent space can follow curved manifolds, but nothing forces neighboring z to decode to similar x away from training data.
- For the VAE, make the encoder output a mean and variance, giving .
- Maximize the ELBO: the first term rewards reconstruction, the KL term pulls each posterior toward N(0,I) so the aggregate latent distribution is smooth and samplable.
- For a Gaussian posterior the KL has the closed form .
Worked example
A VAE encoder outputs, for one input, and for a 2-D latent space. Compute the KL divergence to .
- Use KL = 1/2 sum_j (mu_j^2 + sigma_j^2 - ln sigma_j^2 - 1).
- Dimension 1: 1 + 1 - ln(1) - 1 = 1.
- Dimension 2: 0 + 0.25 - ln(0.25) - 1 = 0.25 + 1.3863 - 1 = 0.6363.
- Sum = 1.6363; multiply by 1/2 = 0.818.
Answer KL ≈ 0.82 nats
Common mistakes
- Assuming any autoencoder latent space is smooth and can be sampled for generation; only regularized ones such as VAEs are designed for this.
- Choosing a latent dimension larger than the data's intrinsic dimension without regularization, so the network learns the identity map.
- Interpreting individual latent axes as independent factors without disentanglement constraints.
- Forgetting the reparameterization trick, so gradients cannot flow through sampling.
- Believing a low reconstruction error implies a useful latent space for downstream tasks.