Content
Summary
A p-value is the probability, assuming the null hypothesis is true, of getting data at least as extreme as what we observed. It measures how surprising the data are under the null, not the probability that the null is true. Small p-values signal data that are hard to explain by chance alone.
Intuition
Imagine a world where the drug does nothing. Even in that world, random variation means sample results will bounce around zero. We build that world's distribution of test statistics, then see where our actual result lands. If it lands far out in the tail, the null world rarely produces results like ours, so we doubt that world. The p-value is the tail area beyond our result. It answers: if nothing is going on, how often would chance alone produce something this extreme or more? It does not tell us how likely the null is, how big the effect is, or how important it is clinically.
Key equations
One-sided p-value: probability under the null that the test statistic is at least as large as observed.
Two-sided p-value for a standard normal test statistic.
Z statistic: how many standard errors the sample mean is from the null mean.
Decision rule with pre-chosen significance level , often 0.05.
Derivation
- State the null hypothesis , for example the mean change in blood pressure is .
- Choose a test statistic that measures departure from , such as .
- Work out its sampling distribution assuming is true; for a known and large this is standard normal.
- Compute the observed value z_obs from the data.
- Find the area in the tail(s) of the null distribution beyond z_obs; that area is the p-value.
- For a two-sided test, count both tails, so double the one-tail area when the distribution is symmetric.
- Compare p with alpha only as a decision aid, and report the effect size and confidence interval too.
Worked example
A drug is claimed to change mean systolic blood pressure. Under the mean change is 0. In patients the mean change is -5 mmHg, with known mmHg. Find the two-sided p-value.
- Standard error = 15/sqrt(36) = 15/6 = 2.5.
- z = (-5 - 0)/2.5 = -2.0.
- One tail: P(Z <= -2.0) is about 0.0228.
- Two-sided p = 2 x 0.0228 = 0.0456.
- Since 0.0456 < 0.05, we reject H0 at the 5% level.
Answer p ≈ 0.046: if the drug truly did nothing, about 4.6% of studies would show a mean change this far from zero or farther.
Common mistakes
- Saying the p-value is the probability that the null hypothesis is true.
- Thinking p = 0.03 means a 97% chance the effect is real.
- Treating a nonsignificant p-value as proof of no effect.
- Equating a small p-value with a large or clinically important effect.
- Choosing or the one-sided test after seeing the data.
- Ignoring multiple testing, which inflates the chance of small p-values by luck.