Content
Summary
Bootstrap resampling estimates the uncertainty of a statistic by repeatedly drawing samples with replacement from the observed data. Each resample yields a new value of the statistic, and the spread of these values approximates its sampling distribution. It gives standard errors and confidence intervals without relying on closed-form formulas.
Intuition
We would love to repeat our study many times and see how much the result varies, but we only have one sample. The bootstrap treats the sample as a stand-in for the population: if the sample is representative, drawing from it with replacement mimics drawing new samples from the population. Each resample has the same size as the original, but some patients appear twice or more and others not at all. Computing the statistic (mean, median, correlation, AUC) on each resample shows how much it wobbles due to sampling alone. The histogram of these values is the bootstrap distribution, and its width estimates the standard error.
Key equations
Statistic T computed on the b-th resample drawn with replacement from the original data of size n.
Bootstrap standard error is the standard deviation of the resampled statistics.
Percentile confidence interval: the α/2 and 1−α/2 quantiles of the bootstrap distribution.
About 37% of the original observations are absent from any given resample.
Derivation
- Observed data x1..xn come from unknown distribution F; the empirical distribution F-hat puts mass 1/n on each observation.
- The sampling distribution of T under F is unknown, so we approximate it by the distribution of T under F-hat.
- Sampling from F-hat is the same as choosing n observations from the data with replacement, each with probability 1/n.
- Repeat B times (e.g., 2000) to get values of the statistic.
- The variability of these values estimates the true sampling variability; take their standard deviation as SE, or their quantiles as a confidence interval.
- The probability a given observation is missed in n draws is (1-1/n)^n, which tends to e^-1, explaining the ~63% unique observations per resample.
Worked example
Ten patients have reductions in systolic blood pressure (mmHg): 4, 7, 9, 3, 12, 6, 8, 5, 10, 6. Describe how to get a bootstrap 95% CI for the mean, and interpret a result where B=1000 resample means have 2.5th and 97.5th percentiles of 5.3 and 8.1.
- Original mean = (4+7+9+3+12+6+8+5+10+6)/10 = 70/10 = 7.0 mmHg.
- Draw 10 values with replacement from the data, compute the mean; repeat 1000 times.
- Sort the 1000 means; take the 25th and 975th values as the 2.5% and 97.5% percentiles.
- These are 5.3 and 8.1, so the interval is (5.3, 8.1).
- Approximate SE from the width: (8.1−5.3)/(2×1.96) ≈ 0.71 mmHg.
Answer Mean 7.0 mmHg with bootstrap 95% CI (5.3, 8.1) mmHg; SE about 0.71.
Common mistakes
- Resampling without replacement, which just reproduces the original sample.
- Using resamples of a different size than n, which changes the variability.
- Using too few resamples (e.g., 20); use at least 1000 for intervals.
- Resampling individual observations when data are clustered or repeated-measures; resample whole patients instead.
- Believing the bootstrap fixes a small or biased sample; it only reflects the data you have.
- Applying it to extreme statistics such as the maximum, where it fails.