Content
Summary
Empirical Bayes (EB) is a computational shortcut for hierarchical Bayesian models that estimates hyperparameters by maximizing the marginal likelihood, effectively approximating the full posterior with a point estimate. This approach facilitates parameter sharing across groups, leading to shrinkage estimators that regularize estimates for groups with small sample sizes by pulling them toward the global mean.
Intuition
Think of Empirical Bayes as a pragmatic compromise between ignoring prior information entirely and fully integrating over all uncertainties. In a hierarchical model, you have local parameters (like a specific group's mean) and hyperparameters (like the global mean and variance). Instead of computing the complex joint posterior for both, EB first looks at the data to find the best hyperparameters that explain the overall distribution. It then treats these hyperparameters as fixed truths to compute the local estimates. This 'borrowing strength' means that if a group has little data, its estimate is heavily influenced by the global pattern, preventing wild swings due to noise. It is computationally efficient and shares information across all groups, though it technically violates the strict Bayesian rule that priors should be independent of the data.
Key equations
The Empirical Bayes estimate of hyperparameters eta, obtained by maximizing the marginal likelihood (evidence).
The shrinkage factor in the Gaussian-Gaussian model, determining the weight given to the global mean versus the local mean.
The posterior mean estimate for group , combining the global mean and local mean based on the shrinkage factor.
The variance of the local observation for group , derived from the base variance and sample size .
Derivation
- Start with the joint posterior distribution in a hierarchical model: .
- Recognize that computing the full posterior over hyperparameters is complex, so approximate with a point estimate .
- Find by maximizing the marginal likelihood (evidence) .
- Use the estimated hyperparameters to compute the local parameter estimates, such as the posterior mean .
- In the Gaussian-Gaussian case, derive the shrinkage factor based on the ratio of local variance to hyperparameter variance .
- Combine the global mean and local mean using the shrinkage factor to form the final estimate.
Worked example
Consider a Gaussian-Gaussian model where you have two groups. Group 1 has a large sample size (), resulting in a small local variance . Group 2 has a small sample size (), resulting in a large local variance . The hyperparameter variance is . Compare the shrinkage factors and and explain how the final estimates and differ in terms of reliance on local data vs. global mean.
- Calculate the shrinkage factor for Group 1: . Since is large, is small. Thus, approaches 0.
- Calculate the shrinkage factor for Group 2: . Since is small, is large. Thus, approaches 1.
- Apply the posterior mean formula: .
- For Group 1, since , . The estimate relies heavily on the local data.
- For Group 2, since , . The estimate is dominated by the global mean.
Answer Group 1's estimate is close to its local mean because the large sample size makes the local data highly reliable. Group 2's estimate is pulled strongly toward the global mean because the small sample size creates high uncertainty, so the model 'borrows strength' from the global population.
Common mistakes
- Empirical Bayes violates the strict Bayesian principle that priors should be chosen independently of the data, as hyperparameters are estimated from the data itself.
- Confusing Empirical Bayes (ML-II) with MAP-II: ML-II maximizes the marginal likelihood, while MAP-II includes a prior on the hyperparameters.
- Assuming that high local MLEs always lead to high posterior estimates: In EB, high MLEs with high uncertainty (small sample size) are shrunk significantly toward the global mean if they conflict with the prior.
- Thinking that Empirical Bayes is identical to Full Bayes: EB uses a point estimate for hyperparameters, whereas Full Bayes integrates over the posterior distribution of hyperparameters.