Biostatistics · Introductory

Confidence interval coverage

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

A 95% confidence interval is produced by a procedure that, over many repeated studies, captures the true population parameter about 95% of the time. Any single interval either contains the true value or it does not. Coverage describes the long-run success rate of the method, not the probability for one interval.

Intuition

Imagine the true mean blood pressure of a population is a fixed but unknown number. You draw a sample, compute a mean and a margin of error, and get an interval. Another researcher draws a different sample and gets a slightly different interval. Because samples vary, the intervals bounce around the fixed true value like horseshoes thrown at a stake. The method is designed so that about 95 out of 100 horseshoes land around the stake. In the simulation, we repeat the study many times, draw each interval as a horizontal bar, and color it green if it covers the true mean and red if it misses. The fraction of green bars approaches 95% as the number of repetitions grows. Small samples with the wrong method, or skewed data, can make actual coverage fall below the nominal level.

Key equations

x‾±tn−1,0.975sn
(1)

95% confidence interval for a population mean from a sample of size n with sample mean x-bar and standard deviation s.

P(L≤μ≤U)=0.95
(2)

Before data are collected, the random interval [L, U] covers the fixed mean μ with probability 0.95.

coverage=#{intervals containing μ}#{intervals}
(3)

Simulated coverage: the proportion of repeated intervals that contain the true value.

SE=sn
(4)

Standard error, which measures how much sample means vary from sample to sample.

Derivation

  1. Start with the sampling distribution: for a Normal population, the quantity (x‾−μ) divided by (s/n) follows a t distribution with n−1 degrees of freedom.
  2. Choose the critical value t* so that 95% of that distribution lies between minus t* and plus t*.
  3. Write the probability statement: P(−t∗≤x‾−μSE≤t∗)=0.95.
  4. Rearrange the inequality to isolate μ: x‾−t∗⋅SE≤μ≤x‾+t∗⋅SE.
  5. The endpoints depend on the random sample, so the interval is random while μ is fixed. Hence 95% of such intervals contain μ in repeated sampling.

Worked example

Problem

A sample of n = 25 patients has mean systolic blood pressure 128 mmHg and standard deviation 15 mmHg. Compute the 95% CI for the population mean. (t with 24 df, 0.975 quantile = 2.064.)

  1. Compute the standard error: SE = 15 / 25 = 15 / 5 = 3.
  2. Compute the margin of error: 2.064 × 3 = 6.19.
  3. Lower limit: 128 - 6.19 = 121.81.
  4. Upper limit: 128 + 6.19 = 134.19.

Answer 95% CI is approximately 121.8 to 134.2 mmHg.

Common mistakes

  • Saying there is a 95% probability that the true mean lies in this particular computed interval; the true mean is fixed, and this interval either contains it or not.
  • Thinking 95% of individual patient values fall within the interval; it concerns the mean, not individuals.
  • Believing that a narrower interval means higher confidence; at fixed confidence, narrower comes from larger n or less variability.
  • Assuming that non-overlapping or overlapping intervals from two studies automatically decide significance.
  • Ignoring that coverage can fall below 95% for small, skewed samples or biased designs.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(114 lines)
import sys
import numpy as np
from manim import *
from blocks import *


def generate_samples(n_samples=50, n_per_sample=25, mu=100, sigma=15):
    """Generate samples and compute CIs. Returns list of (mean, ci_lower, ci_upper, covers)."""
    results = []
    for _ in range(n_samples):
        sample = np.random.normal(mu, sigma, n_per_sample)
        mean = np.mean(sample)
        se = sigma / np.sqrt(n_per_sample)
        ci_lower = mean - 1.96 * se
        ci_upper = mean + 1.96 * se
        covers = ci_lower <= mu <= ci_upper
        results.append((mean, ci_lower, ci_upper, covers))
    return results


N_SAMPLES = 40
N_PER_SAMPLE = 25
MU = 100
SIGMA = 15
np.random.seed(42)
TRACE = generate_samples(N_SAMPLES, N_PER_SAMPLE, MU, SIGMA)


class Main(Scene):
    def construct(self):
        self.camera.background_color = "#111111"
        TitleCard(self, "Confidence Intervals", "95% Coverage Simulation")

        # Setup axes for the plot
        # X axis: Sample Index (0 to N_SAMPLES)
        # Y axis: Value (range around MU)
        # We want to see the interval lines.
        axes = Axes(
            x_range=[0, N_SAMPLES, 5],
            y_range=[MU - 3 * SIGMA / np.sqrt(N_PER_SAMPLE), MU + 3 * SIGMA / np.sqrt(N_PER_SAMPLE), 5],
            x_length=10,
            y_length=6,
            axis_config={"include_tip": False, "color": C_DIM}
        )
        axes.to_edge(LEFT, buff=0.5)
        axes.to_edge(UP, buff=0.5)
        
        x_label = Text("Sample Index", font_size=24, color=WHITE).next_to(axes.x_axis.get_end(), RIGHT, buff=0.2)
        y_label = Text("Value", font_size=24, color=WHITE).next_to(axes.y_axis.get_end(), UP, buff=0.2)
        self.play(Create(axes), FadeIn(x_label), FadeIn(y_label))

        # True mean line
        true_mean_line = DashedLine(
            axes.c2p(0, MU),
            axes.c2p(N_SAMPLES, MU),
            color=C_ACCENT,
            dash_length=0.2
        )
        true_mean_label = Text(f"True Mean ({MU})", font_size=20, color=C_ACCENT)
        true_mean_label.next_to(axes.c2p(N_SAMPLES, MU), RIGHT, buff=0.2)
        self.play(Create(true_mean_line), FadeIn(true_mean_label))

        # Plot intervals
        # We will draw them in batches to keep it readable
        batch_size = 5
        covers_count = 0
        
        for i in range(0, N_SAMPLES, batch_size):
            batch = TRACE[i:i+batch_size]
            anims = []
            for j, (mean, ci_low, ci_high, covers) in enumerate(batch):
                idx = i + j
                # Draw vertical line for CI
                start_pt = axes.c2p(idx, ci_low)
                end_pt = axes.c2p(idx, ci_high)
                
                # Color based on coverage
                color = C_GOOD if covers else C_BAD
                if covers:
                    covers_count += 1
                
                line = Line(start_pt, end_pt, color=color, stroke_width=3)
                # Draw dot at mean
                dot = Dot(axes.c2p(idx, mean), color=color, radius=0.05)
                anims.append(Create(line))
                anims.append(FadeIn(dot))
            
            self.play(*anims, run_time=1.0)
        
        # Update caption with final stats
        coverage_rate = covers_count / N_SAMPLES
        StepCaption(self, f"Coverage: {covers_count}/{N_SAMPLES} = {coverage_rate:.0%}")
        
        self.wait(2)


if __name__ == "__main__" and "--selftest" in sys.argv:
    # Check that we generated the correct number of samples
    assert len(TRACE) == N_SAMPLES
    
    # Check that all CIs are centered around the sample mean
    for mean, ci_low, ci_high, covers in TRACE:
        assert abs((ci_low + ci_high) / 2 - mean) < 1e-9
        
    # Check that CIs are valid (lower < upper)
    for mean, ci_low, ci_high, covers in TRACE:
        assert ci_low < ci_high
        
    # Check that the true mean is within the range of possible CIs (roughly)
    all_lows = [t[1] for t in TRACE]
    all_highs = [t[2] for t in TRACE]
    assert min(all_lows) < MU < max(all_highs)
    
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.