Biostatistics · Intermediate

Maximum likelihood estimation

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Maximum likelihood estimation (MLE) chooses the parameter value that makes the observed data most probable under a assumed model. It turns a data-generating model into an optimisation problem: write the likelihood, take the log, differentiate, and solve. MLE underlies logistic regression, survival models and most of modern biostatistics.

Intuition

Imagine you observe 7 patients respond to a drug out of 10. If the true response probability were 0.1, seeing 7 responders would be very surprising; if it were 0.9, also somewhat unlikely; a value near 0.7 makes your data least surprising. The likelihood function flips the usual viewpoint: the data are fixed, and we ask how plausible each parameter value is. The peak of this curve is the maximum likelihood estimate. We usually maximise the log-likelihood instead, because it turns products over independent patients into sums, which are easier to differentiate, and because the log is increasing, so the peak stays in the same place.

Key equations

L(θ)=∏i=1nf(xi∣θ)
(1)

Likelihood of parameter θ for independent observations x1..xn.

ℓ(θ)=log⁡L(θ)=∑i=1nlog⁡f(xi∣θ)
(2)

Log-likelihood: products become sums.

dℓdθ|θ^=0
(3)

Score equation; its solution is the candidate MLE.

Var⁡(θ^)≈1I(θ),I(θ)=−E[d2ℓdθ2]
(4)

Large-sample variance of the MLE is the inverse Fisher information.

Derivation

  1. Model: each patient responds (x=1) with probability p or not (x=0): f(x|p)=p^x(1-p)^(1-x).
  2. For n independent patients with k responders, L(p)=p^k(1-p)^(n-k).
  3. Take logs: ℓ(p)=k log p + (n-k) log(1-p).
  4. Differentiate: dℓ/dp = k/p - (n-k)/(1-p).
  5. Set to zero: k(1-p)=(n-k)p, so k = np.
  6. Solve: p̂ = k/n.
  7. Check the second derivative: -k/p² - (n-k)/(1-p)² < 0, so this is a maximum.
  8. Fisher information is n/(p(1−p)), so the standard error is p^(1−p^)/n.

Worked example

Problem

7 of 10 patients respond to a drug. Find the MLE of the response probability p and its approximate standard error.

  1. Likelihood: L(p)=p7(1−p)3.
  2. Log-likelihood: ℓ(p)=7 log p + 3 log(1-p).
  3. Score: 7/p - 3/(1-p)=0, giving 7(1-p)=3p, so 7=10p.
  4. p̂=0.7.
  5. Standard error = 0.7×0.3/10=0.021≈0.145.

Answer p̂ = 0.70 with standard error about 0.145.

Common mistakes

  • Treating the likelihood as a probability distribution over the parameter; it does not integrate to one.
  • Forgetting independence: the product form only holds for independent observations.
  • Setting the derivative to zero without checking it is a maximum, or ignoring boundary solutions.
  • Assuming the MLE is unbiased; it is only consistent and asymptotically efficient (for example the normal variance MLE divides by n).
  • Trusting the normal approximation for standard errors with very small samples.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(115 lines)
import sys
import numpy as np
from manim import *
from blocks import *

# Parameters
N_SAMPLES = 20
TRUE_MEAN = 5.0
TRUE_SD = 1.5

def log_likelihood(mu, data, sigma=TRUE_SD):
    """Compute log-likelihood for normal distribution with fixed sigma."""
    n = len(data)
    const = -n/2 * np.log(2 * np.pi) - n * np.log(sigma)
    sum_sq = np.sum((data - mu)**2)
    return const - sum_sq / (2 * sigma**2)

def generate_data():
    """Generate synthetic data from N(TRUE_MEAN, TRUE_SD)."""
    return np.random.default_rng(42).normal(TRUE_MEAN, TRUE_SD, N_SAMPLES)

class Main(Scene):
    def construct(self):
        # Title
        TitleCard(self, "Maximum Likelihood Estimation", "Finding the best fit for observed data")

        # 1. Generate Data
        data = generate_data()
        
        # Display data points as a dot plot
        dot_plot = VGroup()
        for i, val in enumerate(data):
            dot = Dot(point=[val, 0, 0], color=C_MAIN, radius=0.08)
            dot_plot.add(dot)
        dot_plot.move_to([0, 2.5, 0])
        
        # Animate data appearing
        self.play(Create(dot_plot), run_time=2)
        
        # Label the data
        lbl_data = Text("Observed Data (n=20)", font_size=24, color=WHITE)
        lbl_data.next_to(dot_plot, UP, buff=0.2)
        self.play(Write(lbl_data), run_time=1)
        
        # 2. Define Likelihood Function
        # We will plot Log-Likelihood vs Mu
        mu_range = np.linspace(2, 8, 100)
        ll_values = [log_likelihood(mu, data) for mu in mu_range]
        
        # Normalize LL values for better visualization (shift max to 0)
        max_ll = max(ll_values)
        ll_norm = [ll - max_ll for ll in ll_values]
        
        # Create CurvePlot for Log-Likelihood
        # Note: CurvePlot expects a function, but we have precomputed points.
        # We can use a lambda that interpolates or just plot the function directly if we pass the data.
        # Let's use the function directly for cleaner code, but we need to pass data to it.
        # Since CurvePlot takes a function of one variable, we use a closure or partial.
        
        plot = CurvePlot(
            lambda mu: log_likelihood(mu, data),
            x_range=(2, 8, 1),
            y_range=(min(ll_values)-1, max_ll+1, 1),
            x_label="Hypothesized Mean ($\\mu$)",
            y_label="Log-Likelihood",
            color=C_ACCENT,
            x_length=8.0,
            y_length=3.5
        )
        plot.move_to([0, -0.5, 0])
        
        # Animate axes and curve
        self.play(Create(plot.axes), Write(plot.labels), run_time=1.5)
        self.play(Create(plot.curve), run_time=2)
        
        # 3. Find MLE
        # The MLE is the mu that maximizes the likelihood.
        # For normal distribution, MLE of mean is sample mean.
        mle_mu = np.mean(data)
        mle_ll = log_likelihood(mle_mu, data)
        
        # Mark the peak on the graph
        peak_dot = plot.dot_at(mle_mu, color=C_GOOD)
        self.play(FadeIn(peak_dot), run_time=1)
        
        # Label the MLE
        lbl_mle = MathTex(f"\\hat{{\\mu}} = {mle_mu:.2f}", font_size=30, color=C_GOOD)
        lbl_mle.next_to(peak_dot, UP, buff=0.2)
        self.play(Write(lbl_mle), run_time=1)
        
        # 4. Explanation
        eq = MathTex(r"\hat{\mu}_{MLE} = \frac{1}{N} \sum_{i=1}^{N} x_i", font_size=32, color=WHITE)
        eq.to_edge(UP, buff=0.8)
        self.play(Write(eq), run_time=1.5)
        
        StepCaption(self, "The MLE is the parameter value that makes the observed data most probable.")
        self.wait(2)

if __name__ == "__main__" and "--selftest" in sys.argv:
    # Test log likelihood calculation
    test_data = np.array([1.0, 2.0, 3.0])
    ll_at_mean = log_likelihood(2.0, test_data)
    ll_at_other = log_likelihood(3.0, test_data)
    assert ll_at_mean > ll_at_other, "Likelihood should be higher at true mean"
    
    # Test MLE property: mean of data maximizes likelihood
    data = generate_data()
    mle_mu = np.mean(data)
    ll_mle = log_likelihood(mle_mu, data)
    ll_offset = log_likelihood(mle_mu + 1.0, data)
    assert ll_mle > ll_offset, "MLE should have higher likelihood than offset"
    
    # Test that MLE of mean is indeed the sample mean for normal dist
    assert abs(mle_mu - np.mean(data)) < 1e-9
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.