Content
Summary
The Dirichlet-multinomial model generalizes binary inference to K mutually exclusive outcomes by placing a Dirichlet prior over the probability simplex. This conjugate prior allows posterior parameters to be updated by simply adding observed counts to prior pseudo-counts, facilitating straightforward Bayesian inference. The resulting model provides smoother probability estimates and better generalization by accounting for parameter uncertainty.
Intuition
Think of the Dirichlet-multinomial model as an extension of the beta-binomial framework to handle multiple categories, such as the faces of a K-sided die or tokens in text. The core advantage lies in the Dirichlet prior's conjugacy to the multinomial likelihood, which means the posterior distribution remains in the same family as the prior. This allows us to update our beliefs about category probabilities by simply adding the observed frequencies to our initial 'pseudo-counts'. This additive property not only simplifies computation but also acts as a smoothing mechanism, preventing zero probabilities for unseen events and reducing overfitting compared to simple empirical frequency estimates.
Key equations
The likelihood of the data given parameters , where is the count of outcome .
The Dirichlet prior distribution with hyperparameters .
The posterior distribution, where prior parameters are updated by adding observed counts.
The MAP estimate for parameter k, where N is total observations and alpha_0 is the sum of prior hyperparameters.
The MLE recovered when using a uniform prior (alpha_k = 1).
Derivation
- Start with the multinomial likelihood for K outcomes: p(D|θ) = ∏ θ_k^{N_k}, where N_k are the observed counts.
- Place a Dirichlet prior over the parameter vector θ: Dir(θ|α) ∝ ∏ θ_k^{α_k−1}.
- Multiply the likelihood by the prior to obtain the unnormalized posterior: p(θ|D) ∝ ∏ θ_k^{N_k} * ∏ θ_k^{α_k−1} = ∏ θ_k^{N_k + α_k − 1}.
- Recognize that the resulting form is a Dirichlet distribution with updated parameters α_k + N_k.
- To find the Maximum A Posteriori (MAP) estimate, maximize the log-posterior subject to the constraint ∑ θ_k = 1 using a Lagrange multiplier.
- Solve the optimization to derive the MAP estimate: θ̂_k = (N_k + α_k − 1) / (N + α_0 − K).
- Note that if a uniform prior is used (α_k = 1), the MAP estimate simplifies to the Maximum Likelihood Estimate (MLE): θ̂_k = N_k / N.
Worked example
Consider a 3-sided die (K=3). You observe N=10 rolls with counts N=[5, 3, 2]. You use a uniform prior where α_k=1 for all k. Calculate the MAP estimate for the probability of the first outcome.
- Identify the parameters: K=3, N_1=5, N=10, α_1=1.
- Calculate α_0, the sum of prior hyperparameters: α_0 = 1+1+1 = 3.
- Apply the MAP formula: θ̂_1 = (N_1 + α_1 - 1) / (N + α_0 - K).
- Substitute values: θ̂_1 = (5 + 1 - 1) / (10 + 3 - 3) = 5 / 10.
- Simplify: θ̂_1 = 0.5.
Answer 0.5
Common mistakes
- Confusing the MLE with the MAP estimate; the MAP includes prior pseudo-counts and subtracts K in the denominator, whereas MLE is just empirical frequency.
- Forgetting that the Dirichlet prior is defined over the probability simplex, not independent real numbers.
- Assuming the constraint theta_k >= 0 needs explicit enforcement in optimization; the gradient structure naturally discourages negative values.
- Applying the beta-binomial predictive formula directly to the K>1 case without recognizing it as the binary special case of the Dirichlet-multinomial framework.