Content
Summary
Maximum likelihood estimation (MLE) chooses the parameter value that makes the observed data most probable under a assumed model. It turns a data-generating model into an optimisation problem: write the likelihood, take the log, differentiate, and solve. MLE underlies logistic regression, survival models and most of modern biostatistics.
Intuition
Imagine you observe 7 patients respond to a drug out of 10. If the true response probability were 0.1, seeing 7 responders would be very surprising; if it were 0.9, also somewhat unlikely; a value near 0.7 makes your data least surprising. The likelihood function flips the usual viewpoint: the data are fixed, and we ask how plausible each parameter value is. The peak of this curve is the maximum likelihood estimate. We usually maximise the log-likelihood instead, because it turns products over independent patients into sums, which are easier to differentiate, and because the log is increasing, so the peak stays in the same place.
Key equations
Likelihood of parameter for independent observations .
Log-likelihood: products become sums.
Score equation; its solution is the candidate MLE.
Large-sample variance of the MLE is the inverse Fisher information.
Derivation
- Model: each patient responds (x=1) with probability p or not (x=0): f(x|p)=p^x(1-p)^(1-x).
- For n independent patients with k responders, L(p)=p^k(1-p)^(n-k).
- Take logs: ℓ(p)=k log p + (n-k) log(1-p).
- Differentiate: dℓ/dp = k/p - (n-k)/(1-p).
- Set to zero: k(1-p)=(n-k)p, so k = np.
- Solve: p̂ = k/n.
- Check the second derivative: -k/p² - (n-k)/(1-p)² < 0, so this is a maximum.
- Fisher information is , so the standard error is .
Worked example
7 of 10 patients respond to a drug. Find the MLE of the response probability p and its approximate standard error.
- Likelihood: .
- Log-likelihood: ℓ(p)=7 log p + 3 log(1-p).
- Score: 7/p - 3/(1-p)=0, giving 7(1-p)=3p, so 7=10p.
- p̂=0.7.
- Standard error = .
Answer p̂ = 0.70 with standard error about 0.145.
Common mistakes
- Treating the likelihood as a probability distribution over the parameter; it does not integrate to one.
- Forgetting independence: the product form only holds for independent observations.
- Setting the derivative to zero without checking it is a maximum, or ignoring boundary solutions.
- Assuming the MLE is unbiased; it is only consistent and asymptotically efficient (for example the normal variance MLE divides by n).
- Trusting the normal approximation for standard errors with very small samples.