Content
Summary
When many hypothesis tests are run, the chance of at least one false positive grows quickly. Bonferroni controls the family-wise error rate by dividing the significance threshold by the number of tests, while the Benjamini-Hochberg procedure controls the false discovery rate, the expected share of false positives among the discoveries, and so has more power.
Intuition
Imagine testing 20,000 genes at a threshold of 0.05. Even if no gene is truly different, about 1,000 will look significant by chance. Bonferroni is a strict gatekeeper: it demands that you be nearly sure that no false positive appears anywhere, so each test must clear a very high bar. That is sensible for a few confirmatory tests, but it throws away many real effects in large screens. FDR control relaxes the goal. It accepts that some discoveries will be false, and promises only that on average a small fraction, say 5%, of your list is wrong. The Benjamini-Hochberg procedure does this by ranking p-values and letting the threshold rise with rank: the smallest p-value faces a strict cutoff, and larger ranks face progressively looser cutoffs. You find the last p-value that falls under this rising line, and reject everything up to it.
Key equations
Per-test threshold for tests that keeps the family-wise error rate at or below .
Probability of at least one false positive, where V is the number of false positives.
Expected proportion of false discoveries among the R rejected hypotheses.
Benjamini-Hochberg condition: the i-th smallest p-value is compared with a threshold that grows with rank i, at target FDR q.
Derivation
- With independent true nulls each tested at level , , so FWER , which approaches 1 as grows.
- For and , FWER = .
- The union bound gives FWER (per-test level) for any dependence. Setting the per-test level to alpha/m makes FWER alpha. This is Bonferroni.
- For FDR, sort the p-values: p(1) ≤ p(2) ≤... ≤ p(m).
- Find the largest k such that p(k) ≤ (k/m) q.
- Reject the null hypotheses for the k smallest p-values. Benjamini and Hochberg proved this gives FDR ≤ q for independent tests (and under positive dependence).
- Since k/m ≥ 1/m, the BH cutoff is never stricter than Bonferroni at the same level, so BH rejects at least as many hypotheses.
Worked example
Ten tests give sorted p-values: 0.001, 0.008, 0.012, 0.030, 0.041, 0.20, 0.35, 0.55, 0.74, 0.90. Apply Bonferroni at and BH at .
- Bonferroni threshold: 0.05/10 = 0.005. Only 0.001 is below it, so 1 rejection.
- BH thresholds (i/10)(0.05): 0.005, 0.010, 0.015, 0.020, 0.025, 0.030, 0.035, 0.040, 0.045, 0.050.
- Compare: i=1: 0.001 ≤ 0.005 yes. i=2: 0.008 ≤ 0.010 yes. i=3: 0.012 ≤ 0.015 yes. i=4: 0.030 ≤ 0.020 no. i=5: 0.041 ≤ 0.025 no. i=6: 0.20 ≤ 0.030 no. Later ranks also fail.
- The largest i that passes is 3, so reject the three smallest p-values.
Answer Bonferroni rejects 1 hypothesis; Benjamini-Hochberg rejects 3.
Common mistakes
- Stopping at the first p-value that fails in BH; you must find the largest k that passes, and everything below it is rejected even if some of them individually fail.
- Thinking an FDR of 5% means each discovery has a 5% chance of being false, or that FDR bounds the chance of any false positive; it is an expected proportion.
- Applying Bonferroni or BH to p-values that were not sorted or not all from the same family of tests.
- Using a corrected threshold after selectively choosing which tests to report; m must count all tests performed.
- Assuming BH is valid under any dependence; it needs independence or positive dependence, otherwise use the Benjamini-Yekutieli variant.