Content
Summary
A Gaussian Mixture Model (GMM) is a probabilistic density estimation technique that assumes data arises from a mixture of K Gaussian distributions, enabling soft clustering with ellipsoidal shapes. Parameters are estimated via the Expectation-Maximization (EM) algorithm, which iteratively computes cluster responsibilities and updates means, covariances, and weights until convergence.
Intuition
Imagine data points were generated by a hidden process where a cluster is first chosen randomly based on its prevalence (weight), and then a point is sampled from that cluster's Gaussian distribution. Since we don't know which cluster generated which point, we use the EM algorithm to infer these 'latent' assignments. Unlike k-means, which forces a point into exactly one cluster, GMMs calculate the probability (responsibility) that a point belongs to each cluster. This allows the model to capture complex, multimodal data structures where clusters have varying sizes, densities, and orientations, represented mathematically by their covariance matrices.
Key equations
The probability density function of a GMM, combining K Gaussian components weighted by .
The responsibility z_{n,k}, representing the probability that instance n belongs to cluster k.
The update rule for the mean of cluster k, calculated as a weighted average of data points.
The update rule for the mixture weight, representing the fraction of total responsibility assigned to cluster k.
Derivation
- The GMM assumes a generative process where each data point is sampled from one of K Gaussian distributions with probability .
- Since cluster assignments are latent variables, maximum likelihood estimation has no closed-form solution, necessitating the EM algorithm.
- In the Expectation step, responsibilities z_{n,k} are computed using Bayes' rule, reflecting the probability of each cluster generating each data point given current parameters.
- In the Maximization step, parameters are updated: means become weighted averages of data, covariances become weighted covariances, and weights become the average responsibility per cluster.
- This iterative process repeats until the likelihood converges, optimizing the fit of the mixture model to the data.
Worked example
Consider a GMM with K=2 clusters. For a specific data point x_n, the current parameters yield unnormalized scores: and . Calculate the responsibility z_{n,1} and explain how this influences the next mean update for cluster 1.
- Calculate the denominator by summing the scores: 0.3 + 0.7 = 1.0.
- Compute z_{n,1} by dividing the score for cluster 1 by the total: 0.3 / 1.0 = 0.3.
- This responsibility z_{n,1}=0.3 indicates a 30% probability that x_n belongs to cluster 1.
- In the next Maximization step, x_n will contribute to the new mean as a weighted average, with a weight proportional to 0.3, rather than a full weight of 1.0 as in hard clustering.
Answer z_{n,1} = 0.3. The point contributes partially to the new mean of cluster 1, reflecting soft assignment.
Common mistakes
- Confusing GMM with k-means: k-means uses hard assignments and only finds cluster centers, while GMM uses soft assignments and finds centers, shapes, and sizes.
- Assuming EM always finds the global optimum: EM can converge to poor local solutions, so multiple initializations are necessary.
- Thinking a single Gaussian can model complex multimodal data: A single Gaussian is often a poor approximation for data with multiple clusters; GMMs are needed for such cases.