AI, machine learning and deep learning · Intermediate

The Dirichlet-multinomial model

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

The Dirichlet-multinomial model generalizes binary inference to K mutually exclusive outcomes by placing a Dirichlet prior over the probability simplex. This conjugate prior allows posterior parameters to be updated by simply adding observed counts to prior pseudo-counts, facilitating straightforward Bayesian inference. The resulting model provides smoother probability estimates and better generalization by accounting for parameter uncertainty.

Intuition

Think of the Dirichlet-multinomial model as an extension of the beta-binomial framework to handle multiple categories, such as the faces of a K-sided die or tokens in text. The core advantage lies in the Dirichlet prior's conjugacy to the multinomial likelihood, which means the posterior distribution remains in the same family as the prior. This allows us to update our beliefs about category probabilities by simply adding the observed frequencies to our initial 'pseudo-counts'. This additive property not only simplifies computation but also acts as a smoothing mechanism, preventing zero probabilities for unseen events and reducing overfitting compared to simple empirical frequency estimates.

Key equations

p(D|θ)=∏k=1KθkNk
(1)

The likelihood of the data D given parameters θ, where Nk is the count of outcome k.

Dir(θ|α)=1B(α)∏k=1Kθkαk−1
(2)

The Dirichlet prior distribution with hyperparameters α.

p(θ|D)=Dir(θ|α1+N1,…,αK+NK)
(3)

The posterior distribution, where prior parameters are updated by adding observed counts.

θ^k=Nk+αk−1N+α0−K
(4)

The MAP estimate for parameter k, where N is total observations and alpha_0 is the sum of prior hyperparameters.

θ^k=NkN
(5)

The MLE recovered when using a uniform prior (alpha_k = 1).

Derivation

  1. Start with the multinomial likelihood for K outcomes: p(D|θ) = ∏ θ_k^{N_k}, where N_k are the observed counts.
  2. Place a Dirichlet prior over the parameter vector θ: Dir(θ|α) ∝ ∏ θ_k^{α_k−1}.
  3. Multiply the likelihood by the prior to obtain the unnormalized posterior: p(θ|D) ∝ ∏ θ_k^{N_k} * ∏ θ_k^{α_k−1} = ∏ θ_k^{N_k + α_k − 1}.
  4. Recognize that the resulting form is a Dirichlet distribution with updated parameters α_k + N_k.
  5. To find the Maximum A Posteriori (MAP) estimate, maximize the log-posterior subject to the constraint ∑ θ_k = 1 using a Lagrange multiplier.
  6. Solve the optimization to derive the MAP estimate: θ̂_k = (N_k + α_k − 1) / (N + α_0 − K).
  7. Note that if a uniform prior is used (α_k = 1), the MAP estimate simplifies to the Maximum Likelihood Estimate (MLE): θ̂_k = N_k / N.

Worked example

Problem

Consider a 3-sided die (K=3). You observe N=10 rolls with counts N=[5, 3, 2]. You use a uniform prior where α_k=1 for all k. Calculate the MAP estimate for the probability of the first outcome.

  1. Identify the parameters: K=3, N_1=5, N=10, α_1=1.
  2. Calculate α_0, the sum of prior hyperparameters: α_0 = 1+1+1 = 3.
  3. Apply the MAP formula: θ̂_1 = (N_1 + α_1 - 1) / (N + α_0 - K).
  4. Substitute values: θ̂_1 = (5 + 1 - 1) / (10 + 3 - 3) = 5 / 10.
  5. Simplify: θ̂_1 = 0.5.

Answer 0.5

Common mistakes

  • Confusing the MLE with the MAP estimate; the MAP includes prior pseudo-counts and subtracts K in the denominator, whereas MLE is just empirical frequency.
  • Forgetting that the Dirichlet prior is defined over the probability simplex, not independent real numbers.
  • Assuming the constraint theta_k >= 0 needs explicit enforcement in optimization; the gradient structure naturally discourages negative values.
  • Applying the beta-binomial predictive formula directly to the K>1 case without recognizing it as the binary special case of the Dirichlet-multinomial framework.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.