Content
Summary
Inference in jointly Gaussian distributions allows for the exact computation of marginal and conditional probabilities, which are themselves Gaussian. The conditional distribution is characterized by a mean that is a linear function of the observed variables and a covariance that is constant and independent of the observations.
Intuition
When dealing with a system of variables that are jointly Gaussian, observing one part of the system provides linear information about the other parts. This means that if you know the value of one variable, you can update your belief about another variable by shifting its mean linearly based on the deviation of the observed value from its prior mean. Crucially, this observation also reduces uncertainty; the new variance (conditional covariance) is always smaller than or equal to the original variance, reflecting the information gained. If the variables are uncorrelated, no information is gained, and the conditional distribution remains identical to the marginal distribution.
Key equations
Marginal distribution of x1, obtained by extracting the mean vector mu1 and covariance submatrix Sigma11 from the joint parameters.
Posterior conditional mean of x1 given x2, using covariance parameters.
Posterior conditional covariance of x1 given x2, representing the remaining uncertainty.
Alternative expression for posterior conditional covariance using the precision matrix block Lambda11.
Derivation
- Start with a jointly Gaussian vector with mean and covariance .
- To find the marginal distribution p(x1), extract the corresponding sub-vector mu1 and sub-matrix Sigma11 from the joint parameters.
- To find the conditional distribution p(x1|x2), recognize that the mean becomes a linear function of x2, specifically shifting by an amount proportional to the deviation (x2 - mu2).
- The conditional covariance is calculated by subtracting the information gained from x2 from the prior covariance, resulting in a constant matrix independent of the specific value of x2.
- Verify that if the correlation rho is zero, the shift term vanishes and the conditional covariance equals the marginal covariance, confirming independence.
Worked example
Consider a 2D Gaussian with equal variances and correlation . If we observe , what is the conditional mean and variance of ?
- Identify the parameters: prior mean , prior variance , and correlation .
- Calculate the mean shift: The conditional mean shifts by rho * (x2 - mu2).
- Calculate the variance reduction: The conditional variance reduces to .
- Interpret the result: Higher correlation leads to a larger mean shift and greater variance reduction.
Answer Conditional Mean: ; Conditional Variance:
Common mistakes
- Confusing the conditional covariance with the marginal covariance; the conditional covariance is generally smaller and does not depend on the specific value of x2.
- Assuming that uncorrelated Gaussian variables are not independent; for Gaussians, zero correlation implies independence, so the conditional equals the marginal.
- Misinterpreting the linear shift in the conditional mean; it depends on the deviation of x2 from its mean, not just the value of x2.