Biostatistics · Intermediate

Bonferroni and FDR control

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

When many hypothesis tests are run, the chance of at least one false positive grows quickly. Bonferroni controls the family-wise error rate by dividing the significance threshold by the number of tests, while the Benjamini-Hochberg procedure controls the false discovery rate, the expected share of false positives among the discoveries, and so has more power.

Intuition

Imagine testing 20,000 genes at a threshold of 0.05. Even if no gene is truly different, about 1,000 will look significant by chance. Bonferroni is a strict gatekeeper: it demands that you be nearly sure that no false positive appears anywhere, so each test must clear a very high bar. That is sensible for a few confirmatory tests, but it throws away many real effects in large screens. FDR control relaxes the goal. It accepts that some discoveries will be false, and promises only that on average a small fraction, say 5%, of your list is wrong. The Benjamini-Hochberg procedure does this by ranking p-values and letting the threshold rise with rank: the smallest p-value faces a strict cutoff, and larger ranks face progressively looser cutoffs. You find the last p-value that falls under this rising line, and reject everything up to it.

Key equations

αBonf=αm
(1)

Per-test threshold for m tests that keeps the family-wise error rate at or below α.

FWER=P(V≥1)
(2)

Probability of at least one false positive, where V is the number of false positives.

FDR=E[Vmax⁡(R,1)]
(3)

Expected proportion of false discoveries among the R rejected hypotheses.

p(i)≤imq
(4)

Benjamini-Hochberg condition: the i-th smallest p-value is compared with a threshold that grows with rank i, at target FDR q.

Derivation

  1. With m independent true nulls each tested at level α, P(no false positive)=(1−α)m, so FWER =1−(1−α)m, which approaches 1 as m grows.
  2. For α=0.05 and m=20, FWER = 1−0.9520≈0.64.
  3. The union bound gives FWER ≤m× (per-test level) for any dependence. Setting the per-test level to alpha/m makes FWER ≤ alpha. This is Bonferroni.
  4. For FDR, sort the p-values: p(1) ≤ p(2) ≤... ≤ p(m).
  5. Find the largest k such that p(k) ≤ (k/m) q.
  6. Reject the null hypotheses for the k smallest p-values. Benjamini and Hochberg proved this gives FDR ≤ q for independent tests (and under positive dependence).
  7. Since k/m ≥ 1/m, the BH cutoff is never stricter than Bonferroni at the same level, so BH rejects at least as many hypotheses.

Worked example

Problem

Ten tests give sorted p-values: 0.001, 0.008, 0.012, 0.030, 0.041, 0.20, 0.35, 0.55, 0.74, 0.90. Apply Bonferroni at α=0.05 and BH at q=0.05.

  1. Bonferroni threshold: 0.05/10 = 0.005. Only 0.001 is below it, so 1 rejection.
  2. BH thresholds (i/10)(0.05): 0.005, 0.010, 0.015, 0.020, 0.025, 0.030, 0.035, 0.040, 0.045, 0.050.
  3. Compare: i=1: 0.001 ≤ 0.005 yes. i=2: 0.008 ≤ 0.010 yes. i=3: 0.012 ≤ 0.015 yes. i=4: 0.030 ≤ 0.020 no. i=5: 0.041 ≤ 0.025 no. i=6: 0.20 ≤ 0.030 no. Later ranks also fail.
  4. The largest i that passes is 3, so reject the three smallest p-values.

Answer Bonferroni rejects 1 hypothesis; Benjamini-Hochberg rejects 3.

Common mistakes

  • Stopping at the first p-value that fails in BH; you must find the largest k that passes, and everything below it is rejected even if some of them individually fail.
  • Thinking an FDR of 5% means each discovery has a 5% chance of being false, or that FDR bounds the chance of any false positive; it is an expected proportion.
  • Applying Bonferroni or BH to p-values that were not sorted or not all from the same family of tests.
  • Using a corrected threshold after selectively choosing which tests to report; m must count all tests performed.
  • Assuming BH is valid under any dependence; it needs independence or positive dependence, otherwise use the Benjamini-Yekutieli variant.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(226 lines)
import sys
from manim import *
from blocks import *
import numpy as np

# --- Helper: Simulation of Multiple Comparisons ---

def run_multiple_comparisons(n_tests, alpha, method="bonferroni"):
    """
    Simulate n_tests independent hypothesis tests.
    True nulls are 50% of tests.
    Returns a list of snapshots: (p_value, is_null, is_rejected, adjusted_alpha)
    """
    np.random.seed(42)
    n_null = int(n_tests * 0.5)
    n_alt = n_tests - n_null
    
    # Generate p-values: Uniform[0,1] for null, Beta(2,10) for alt (shifted low)
    p_nulls = np.random.uniform(0, 1, n_null)
    p_alts = np.random.beta(2, 10, n_alt)
    
    p_values = np.concatenate([p_nulls, p_alts])
    is_null = np.array([True] * n_null + [False] * n_alt)
    
    # Shuffle to mix null/alt
    indices = np.random.permutation(n_tests)
    p_values = p_values[indices]
    is_null = is_null[indices]
    
    trace = []
    if method == "bonferroni":
        adj_alpha = alpha / n_tests
        for i in range(n_tests):
            rejected = p_values[i] < adj_alpha
            trace.append({
                "i": i,
                "p": p_values[i],
                "null": is_null[i],
                "rejected": rejected,
                "adj_alpha": adj_alpha,
                "method": "Bonferroni"
            })
    elif method == "fdr":
        # Benjamini-Hochberg Procedure
        sorted_indices = np.argsort(p_values)
        sorted_p = p_values[sorted_indices]
        sorted_is_null = is_null[sorted_indices]
        
        # Calculate critical values
        m = n_tests
        thresholds = np.arange(1, m + 1) * alpha / m
        
        # Find largest k where p_(k) <= k/m * alpha
        k = -1
        for i in range(m):
            if sorted_p[i] <= thresholds[i]:
                k = i
        
        # Reject all hypotheses with p <= p_(k)
        reject_threshold = sorted_p[k] if k >= 0 else 1.0
        
        # Reconstruct trace in original order
        # We need to know for each original index if it was rejected
        # A simple way: mark based on the global threshold derived from BH
        # Note: Strict BH is step-up, but for visualization, we can show the threshold line
        
        # Let's create a trace that shows the sorted order for clarity in FDR
        # But the prompt asks for a trace. Let's do a simplified "Storey's" or just BH step-by-step?
        # To keep it simple and visual, let's just show the BH sorted list and the cutoff.
        
        # Actually, let's stick to the raw list but highlight the BH logic.
        # Better: Show the sorted p-values and the line.
        
        bh_trace = []
        for i in range(m):
            p = sorted_p[i]
            thresh = thresholds[i]
            rejected = p <= reject_threshold # Simplified final decision
            bh_trace.append({
                "i": i, # Index in sorted list
                "p": p,
                "null": sorted_is_null[i],
                "rejected": rejected,
                "adj_alpha": thresh, # The specific threshold for this rank
                "method": "BH (FDR)",
                "global_cutoff": reject_threshold
            })
        return bh_trace, sorted_indices # Return extra info for plotting

    return trace

class Main(Scene):
    def construct(self):
        TitleCard(self, "Multiple Comparisons", "Bonferroni vs FDR")
        
        n_tests = 20
        alpha = 0.05
        
        # --- Part 1: Bonferroni ---
        StepCaption(self, "1. Bonferroni: Divide alpha by n tests")
        
        bon_trace = run_multiple_comparisons(n_tests, alpha, "bonferroni")
        adj_alpha = bon_trace[0]["adj_alpha"]
        
        # Create a visual representation of the tests
        # We will use a GridView-like structure but custom for p-values
        # Let's use ArrayBars for p-values, colored by rejection
        
        # Initial state: all bars
        p_vals = [t["p"] for t in bon_trace]
        bars = ArrayBars(p_vals, max_width=10, max_height=3)
        bars.to_edge(LEFT, buff=0.5)
        bars.shift(UP * 0.5)
        
        self.play(Create(bars.bars), Create(bars.labels))
        
        # Show the threshold line
        thresh_line = Line(
            start=bars.get_left() + np.array([-0.5, 0, 0]),
            end=bars.get_left() + np.array([10.5, 0, 0]),
            color=C_BAD
        )
        # Position the line at the height corresponding to adj_alpha
        # Max height is 3 for value 1.0. So height for adj_alpha is 3 * adj_alpha
        line_y = -1.5 + 3 * adj_alpha
        thresh_line.shift(UP * line_y)
        
        thresh_label = MathTex(f"\\alpha' = {adj_alpha:.3f}", color=C_BAD).next_to(thresh_line, UP)
        
        self.play(FadeIn(thresh_line), FadeIn(thresh_label))
        
        # Animate rejection
        rejected_indices = [i for i, t in enumerate(bon_trace) if t["rejected"]]
        null_rejected = [i for i in rejected_indices if bon_trace[i]["null"]]
        
        # Highlight rejected bars
        self.play(bars.mark(rejected_indices, C_BAD))
        
        # Count errors
        n_false_pos = len(null_rejected)
        n_total_rej = len(rejected_indices)
        
        error_text = Text(f"False Positives: {n_false_pos}", font_size=24, color=C_BAD)
        error_text.next_to(bars, DOWN, buff=0.5)
        self.play(FadeIn(error_text))
        
        self.wait(1)
        
        # --- Part 2: FDR (BH) ---
        StepCaption(self, "2. FDR (BH): Rank-based threshold")
        
        self.play(FadeOut(bars), FadeOut(thresh_line), FadeOut(thresh_label), FadeOut(error_text))
        
        fdr_trace, sorted_indices = run_multiple_comparisons(n_tests, alpha, "fdr")
        
        # Sort p-values for BH visualization
        sorted_p = [t["p"] for t in fdr_trace]
        sorted_is_null = [t["null"] for t in fdr_trace]
        
        # Create bars for sorted p-values
        fdr_bars = ArrayBars(sorted_p, max_width=10, max_height=3)
        fdr_bars.to_edge(LEFT, buff=0.5)
        fdr_bars.shift(UP * 0.5)
        
        self.play(Create(fdr_bars.bars), Create(fdr_bars.labels))
        
        # Draw the BH line (k/m * alpha)
        # We will draw a step function or just the cutoff line
        # The cutoff is the p-value of the largest k satisfying the condition.
        # Let's find the cutoff index from the trace
        cutoff_p = fdr_trace[-1]["global_cutoff"]
        
        # Draw a horizontal line at the cutoff p-value
        cutoff_y = -1.5 + 3 * cutoff_p
        cutoff_line = Line(
            start=fdr_bars.get_left() + np.array([-0.5, 0, 0]),
            end=fdr_bars.get_left() + np.array([10.5, 0, 0]),
            color=C_GOOD
        )
        cutoff_line.shift(UP * cutoff_y)
        
        cutoff_label = MathTex("FDR Cutoff", color=C_GOOD).next_to(cutoff_line, UP)
        
        self.play(FadeIn(cutoff_line), FadeIn(cutoff_label))
        
        # Highlight rejected bars (those below the cutoff)
        # In BH, all p <= cutoff_p are rejected
        rej_indices_fdr = [i for i, t in enumerate(fdr_trace) if t["rejected"]]
        
        self.play(fdr_bars.mark(rej_indices_fdr, C_GOOD))
        
        # Count errors
        fdr_null_rej = [i for i in rej_indices_fdr if sorted_is_null[i]]
        n_fdr_false_pos = len(fdr_null_rej)
        
        fdr_error_text = Text(f"False Positives: {n_fdr_false_pos}", font_size=24, color=C_GOOD)
        fdr_error_text.next_to(fdr_bars, DOWN, buff=0.5)
        self.play(FadeIn(fdr_error_text))
        
        self.wait(1)
        
        # --- Comparison ---
        StepCaption(self, "Bonferroni is conservative; FDR allows more discoveries")
        
        self.wait(2)

if __name__ == "__main__" and "--selftest" in sys.argv:
    # Test Bonferroni
    bon_trace = run_multiple_comparisons(20, 0.05, "bonferroni")
    assert len(bon_trace) == 20
    assert bon_trace[0]["adj_alpha"] == 0.05 / 20
    
    # Test FDR
    fdr_trace, _ = run_multiple_comparisons(20, 0.05, "fdr")
    assert len(fdr_trace) == 20
    # Check that the last element has the global cutoff
    assert "global_cutoff" in fdr_trace[-1]
    
    # Check that rejected items are below the cutoff
    for t in fdr_trace:
        if t["rejected"]:
            assert t["p"] <= t["global_cutoff"] + 1e-9
        else:
            assert t["p"] > t["global_cutoff"] - 1e-9
            
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.