AI, machine learning and deep learning · Intermediate

Clustering with a Mixture of Gaussians

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

A Gaussian Mixture Model (GMM) is a probabilistic density estimation technique that assumes data arises from a mixture of K Gaussian distributions, enabling soft clustering with ellipsoidal shapes. Parameters are estimated via the Expectation-Maximization (EM) algorithm, which iteratively computes cluster responsibilities and updates means, covariances, and weights until convergence.

Intuition

Imagine data points were generated by a hidden process where a cluster is first chosen randomly based on its prevalence (weight), and then a point is sampled from that cluster's Gaussian distribution. Since we don't know which cluster generated which point, we use the EM algorithm to infer these 'latent' assignments. Unlike k-means, which forces a point into exactly one cluster, GMMs calculate the probability (responsibility) that a point belongs to each cluster. This allows the model to capture complex, multimodal data structures where clusters have varying sizes, densities, and orientations, represented mathematically by their covariance matrices.

Key equations

p(x|θ)=∑k=1Kπk𝒩(x|μk,Σk)
(1)

The probability density function of a GMM, combining K Gaussian components weighted by πk.

zn,k=πk𝒩(xn|μk,Σk)∑j=1Kπj𝒩(xn|μj,Σj)
(2)

The responsibility z_{n,k}, representing the probability that instance n belongs to cluster k.

μk=∑n=1Nzn,kxn∑n=1Nzn,k
(3)

The update rule for the mean of cluster k, calculated as a weighted average of data points.

πk=1N∑n=1Nzn,k
(4)

The update rule for the mixture weight, representing the fraction of total responsibility assigned to cluster k.

Derivation

  1. The GMM assumes a generative process where each data point is sampled from one of K Gaussian distributions with probability πk.
  2. Since cluster assignments are latent variables, maximum likelihood estimation has no closed-form solution, necessitating the EM algorithm.
  3. In the Expectation step, responsibilities z_{n,k} are computed using Bayes' rule, reflecting the probability of each cluster generating each data point given current parameters.
  4. In the Maximization step, parameters are updated: means become weighted averages of data, covariances become weighted covariances, and weights become the average responsibility per cluster.
  5. This iterative process repeats until the likelihood converges, optimizing the fit of the mixture model to the data.

Worked example

Problem

Consider a GMM with K=2 clusters. For a specific data point x_n, the current parameters yield unnormalized scores: π1𝒩(xn|μ1,Σ1)=0.3 and π2𝒩(xn|μ2,Σ2)=0.7. Calculate the responsibility z_{n,1} and explain how this influences the next mean update for cluster 1.

  1. Calculate the denominator by summing the scores: 0.3 + 0.7 = 1.0.
  2. Compute z_{n,1} by dividing the score for cluster 1 by the total: 0.3 / 1.0 = 0.3.
  3. This responsibility z_{n,1}=0.3 indicates a 30% probability that x_n belongs to cluster 1.
  4. In the next Maximization step, x_n will contribute to the new mean μ1 as a weighted average, with a weight proportional to 0.3, rather than a full weight of 1.0 as in hard clustering.

Answer z_{n,1} = 0.3. The point contributes partially to the new mean of cluster 1, reflecting soft assignment.

Common mistakes

  • Confusing GMM with k-means: k-means uses hard assignments and only finds cluster centers, while GMM uses soft assignments and finds centers, shapes, and sizes.
  • Assuming EM always finds the global optimum: EM can converge to poor local solutions, so multiple initializations are necessary.
  • Thinking a single Gaussian can model complex multimodal data: A single Gaussian is often a poor approximation for data with multiple clusters; GMMs are needed for such cases.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.