AI, machine learning and deep learning · Advanced

Autoencoder latent space

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

An autoencoder compresses an input x into a low-dimensional latent vector z through an encoder, then reconstructs x from z with a decoder. Training minimizes reconstruction error, so the latent space learns a compact, task-relevant coordinate system of the data manifold. Its geometry (smoothness, dimensionality, regularization) determines whether interpolation and generation are meaningful.

Intuition

Real data such as images or gene expression profiles do not fill their high-dimensional space; they lie near a low-dimensional curved surface, the manifold. The bottleneck forces the network to discard directions that carry no information for reconstruction and keep the few that do. Think of z as map coordinates: the encoder projects a point in a huge space onto a map, and the decoder lifts it back. A plain autoencoder only guarantees that points from the training set reconstruct well, so the space between them may be 'holes' that decode to nonsense. A variational autoencoder adds a KL penalty that pulls codes toward a Gaussian prior, filling holes and making the space smooth at some cost to reconstruction sharpness. A linear autoencoder with squared error recovers the same subspace as PCA.

Key equations

z=fϕ(x),x^=gθ(z)
(1)

Encoder f maps input to latent code; decoder g maps code back to a reconstruction.

ℒAE=1N∑i=1N‖xi−gθ(fϕ(xi))‖2
(2)

Reconstruction loss minimized over encoder and decoder parameters.

ℒVAE=𝔼qϕ(z|x)[log⁡pθ(x|z)]−DKL(qϕ(z|x)‖𝒩(0,I))
(3)

VAE evidence lower bound: reconstruction term minus a regularizer shaping the latent space.

z=μ+σ⊙ϵ,ϵ∼𝒩(0,I)
(4)

Reparameterization trick allowing gradients through sampling.

Derivation

  1. Assume data x in R^D lies near a d-dimensional manifold with d much smaller than D.
  2. Choose an encoder f mapping R^D to R^d and a decoder g mapping R^d to R^D.
  3. Define the loss as mean squared error between x and g(f(x)); gradient descent updates both networks by backpropagation through the bottleneck.
  4. For linear f=Wx and g=Vz, the optimum has V W as the projection onto the top d principal components, so the latent space spans the PCA subspace (up to an invertible transform).
  5. With nonlinear networks the latent space can follow curved manifolds, but nothing forces neighboring z to decode to similar x away from training data.
  6. For the VAE, make the encoder output a mean and variance, giving q(z|x)=𝒩(μ,diag σ2).
  7. Maximize the ELBO: the first term rewards reconstruction, the KL term pulls each posterior toward N(0,I) so the aggregate latent distribution is smooth and samplable.
  8. For a Gaussian posterior the KL has the closed form 12∑(μ2+σ2−log⁡σ2−1).

Worked example

Problem

A VAE encoder outputs, for one input, μ=(1,0) and σ=(1,0.5) for a 2-D latent space. Compute the KL divergence to 𝒩(0,I).

  1. Use KL = 1/2 sum_j (mu_j^2 + sigma_j^2 - ln sigma_j^2 - 1).
  2. Dimension 1: 1 + 1 - ln(1) - 1 = 1.
  3. Dimension 2: 0 + 0.25 - ln(0.25) - 1 = 0.25 + 1.3863 - 1 = 0.6363.
  4. Sum = 1.6363; multiply by 1/2 = 0.818.

Answer KL ≈ 0.82 nats

Common mistakes

  • Assuming any autoencoder latent space is smooth and can be sampled for generation; only regularized ones such as VAEs are designed for this.
  • Choosing a latent dimension larger than the data's intrinsic dimension without regularization, so the network learns the identity map.
  • Interpreting individual latent axes as independent factors without disentanglement constraints.
  • Forgetting the reparameterization trick, so gradients cannot flow through sampling.
  • Believing a low reconstruction error implies a useful latent space for downstream tasks.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(193 lines)
import sys
import numpy as np
from manim import *
from blocks import *

# --- Simulation Logic ---

def encoder(x):
    """Simple linear encoder: x -> [x, x^2/4]. Maps input to 2D latent."""
    return np.array([x, x**2 / 4.0])

def decoder(z):
    """Simple linear decoder: [z0, z1] -> z0 (ignoring z1 for reconstruction)."""
    return z[0]

def autoencoder_trace(n_points=15, noise_std=0.05):
    """
    Simulate an autoencoder training run.
    We assume the model has learned to reconstruct x from z.
    We trace the mapping of input points through the latent space.
    """
    # Ground truth inputs
    inputs = np.linspace(-3.0, 3.0, n_points)
    # Add small noise to simulate real data distribution
    noisy_inputs = inputs + np.random.RandomState(42).normal(0, noise_std, n_points)
    
    trace = []
    for x in noisy_inputs:
        z = encoder(x)
        x_rec = decoder(z)
        trace.append({
            "x": x,
            "z": z,
            "x_rec": x_rec,
            "error": abs(x - x_rec)
        })
    return trace

TRACE = autoencoder_trace()

class Main(Scene):
    def construct(self):
        self.camera.background_color = "#101418"
        
        # Title
        TitleCard(self, "Autoencoder Latent Space", "Compression and Reconstruction")
        
        # Layout Setup
        # Left: Input Space (1D line)
        # Center: Latent Space (2D plane)
        # Right: Reconstructed Space (1D line)
        
        input_space = GraphView(
            nodes=["x"],
            edges=[],
            layout={"x": np.array([-4.0, 0, 0])},
            max_width=3.0, max_height=5.0
        ).shift(LEFT * 4.5)
        
        latent_space = GraphView(
            nodes=["z1", "z2"],
            edges=[("z1", "z2")],
            layout={
                "z1": np.array([-1.5, -1.5, 0]),
                "z2": np.array([1.5, 1.5, 0])
            },
            max_width=4.0, max_height=5.0
        ).shift(ORIGIN)
        
        recon_space = GraphView(
            nodes=["x'"],
            edges=[],
            layout={"x'": np.array([4.0, 0, 0])},
            max_width=3.0, max_height=5.0
        ).shift(RIGHT * 4.5)
        
        # Labels for spaces
        lbl_input = Text("Input", font_size=24, color=C_DIM).next_to(input_space, UP)
        lbl_latent = Text("Latent", font_size=24, color=C_DIM).next_to(latent_space, UP)
        lbl_recon = Text("Reconstruction", font_size=24, color=C_DIM).next_to(recon_space, UP)
        
        self.play(FadeIn(input_space), FadeIn(latent_space), FadeIn(recon_space))
        self.play(FadeIn(lbl_input), FadeIn(lbl_latent), FadeIn(lbl_recon))
        
        # Draw axes for latent space to make it look like a plot
        axes_latent = Axes(
            x_range=[-3, 3, 1], y_range=[-2, 3, 1],
            x_length=3.5, y_length=3.5,
            axis_config={"color": C_DIM}
        ).move_to(latent_space.get_center())
        
        # We will animate points moving from Input -> Latent -> Recon
        # To visualize the "trace", we show a few representative points
        
        # Helper to create a dot at a specific logical position
        # Input is 1D, so we map x to the line at x=-4
        # Latent is 2D, we map z to the graph coordinates
        # Recon is 1D, we map x' to the line at x=4
        
        # Since GraphView nodes are fixed, we need to place dots relative to them or use custom positions.
        # Let's use custom positions for clarity.
        
        # Input Line
        input_line = Line(start=np.array([-4.0, -2.5, 0]), end=np.array([-4.0, 2.5, 0]), color=C_DIM)
        # Recon Line
        recon_line = Line(start=np.array([4.0, -2.5, 0]), end=np.array([4.0, 2.5, 0]), color=C_DIM)
        
        self.play(Create(input_line), Create(recon_line))
        
        # Animate a subset of points to keep it clean
        # Pick 5 points spread across the range
        indices = [0, 3, 7, 11, 14]
        
        dots_input = []
        dots_latent = []
        dots_recon = []
        
        for i in indices:
            data = TRACE[i]
            x_val = data["x"]
            z_val = data["z"]
            x_rec = data["x_rec"]
            
            # Map logical values to screen coordinates
            # Input: x in [-3, 3] -> y in [-2.5, 2.5]
            y_in = (x_val / 3.0) * 2.5
            pos_in = np.array([-4.0, y_in, 0])
            
            # Latent: z in [-3, 3] x [-2, 3] -> screen [-1.75, 1.75] x [-1.75, 1.75]
            # Note: latent_space graph is centered at 0,0. 
            # We need to map z coordinates to the axes we just drew or the graph bounds.
            # Let's use the axes coordinates directly.
            # z[0] is z1 (x-axis), z[1] is z2 (y-axis)
            # But our axes range is x:[-3,3], y:[-2,3].
            # Let's scale to fit the 3.5x3.5 box.
            z1_screen = (z_val[0] / 3.0) * 1.75
            z2_screen = (z_val[1] / 3.0) * 1.75 # Approximate scaling for visualization
            pos_lat = np.array([z1_screen, z2_screen, 0])
            
            # Recon: x' in [-3, 3] -> y in [-2.5, 2.5]
            y_rec = (x_rec / 3.0) * 2.5
            pos_rec = np.array([4.0, y_rec, 0])
            
            dot_in = Dot(pos_in, color=C_MAIN, radius=0.1)
            dot_lat = Dot(pos_lat, color=C_ACCENT, radius=0.1)
            dot_rec = Dot(pos_rec, color=C_GOOD, radius=0.1)
            
            dots_input.append(dot_in)
            dots_latent.append(dot_lat)
            dots_recon.append(dot_rec)
            
            # Initial state: all at input
            self.add(dot_in)
            
        # Step 1: Show Input Points
        StepCaption(self, "1. Input data points (noisy)")
        self.wait(0.5)
        
        # Step 2: Encode to Latent
        anims_enc = [d.animate.move_to(dl) for d, dl in zip(dots_input, dots_latent)]
        self.play(*anims_enc, run_time=1.5)
        StepCaption(self, "2. Encoder compresses to latent vector z")
        self.wait(0.5)
        
        # Step 3: Decode to Reconstruction
        anims_dec = [d.animate.move_to(dr) for d, dr in zip(dots_latent, dots_recon)]
        self.play(*anims_dec, run_time=1.5)
        StepCaption(self, "3. Decoder reconstructs x' from z")
        self.wait(0.5)
        
        # Step 4: Highlight Error
        # Show lines connecting input to recon to visualize error
        error_lines = VGroup()
        for d_in, d_rec in zip(dots_input, dots_recon):
            line = Line(d_in.get_center(), d_rec.get_center(), color=C_BAD, stroke_width=1, stroke_opacity=0.5)
            error_lines.add(line)
        
        self.play(Create(error_lines), run_time=1.0)
        StepCaption(self, "4. Reconstruction error (distance between x and x')")
        self.wait(1)

if __name__ == "__main__" and "--selftest" in sys.argv:
    # Check that reconstruction is close to input for this simple linear autoencoder
    for item in TRACE:
        # The decoder is just identity on the first component, encoder is [x, x^2/4]
        # So x_rec should be x.
        assert abs(item["x_rec"] - item["x"]) < 1e-10, f"Reconstruction failed for x={item['x']}"
    
    # Check latent shape
    for item in TRACE:
        assert len(item["z"]) == 2, "Latent vector should be 2D"
        
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.