Biostatistics · Introductory

P-value under the null

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

A p-value is the probability, assuming the null hypothesis is true, of getting data at least as extreme as what we observed. It measures how surprising the data are under the null, not the probability that the null is true. Small p-values signal data that are hard to explain by chance alone.

Intuition

Imagine a world where the drug does nothing. Even in that world, random variation means sample results will bounce around zero. We build that world's distribution of test statistics, then see where our actual result lands. If it lands far out in the tail, the null world rarely produces results like ours, so we doubt that world. The p-value is the tail area beyond our result. It answers: if nothing is going on, how often would chance alone produce something this extreme or more? It does not tell us how likely the null is, how big the effect is, or how important it is clinically.

Key equations

p=P(T≥tobs∣H0)
(1)

One-sided p-value: probability under the null that the test statistic is at least as large as observed.

p=2P(Z≥|zobs|)
(2)

Two-sided p-value for a standard normal test statistic.

z=x‾−μ0σ/n
(3)

Z statistic: how many standard errors the sample mean is from the null mean.

Reject H0 if p≤α
(4)

Decision rule with pre-chosen significance level α, often 0.05.

Derivation

  1. State the null hypothesis H0, for example the mean change in blood pressure is μ0=0.
  2. Choose a test statistic that measures departure from H0, such as z.
  3. Work out its sampling distribution assuming H0 is true; for a known σ and large n this is standard normal.
  4. Compute the observed value z_obs from the data.
  5. Find the area in the tail(s) of the null distribution beyond z_obs; that area is the p-value.
  6. For a two-sided test, count both tails, so double the one-tail area when the distribution is symmetric.
  7. Compare p with alpha only as a decision aid, and report the effect size and confidence interval too.

Worked example

Problem

A drug is claimed to change mean systolic blood pressure. Under H0 the mean change is 0. In n=36 patients the mean change is -5 mmHg, with known σ=15 mmHg. Find the two-sided p-value.

  1. Standard error = 15/sqrt(36) = 15/6 = 2.5.
  2. z = (-5 - 0)/2.5 = -2.0.
  3. One tail: P(Z <= -2.0) is about 0.0228.
  4. Two-sided p = 2 x 0.0228 = 0.0456.
  5. Since 0.0456 < 0.05, we reject H0 at the 5% level.

Answer p ≈ 0.046: if the drug truly did nothing, about 4.6% of studies would show a mean change this far from zero or farther.

Common mistakes

  • Saying the p-value is the probability that the null hypothesis is true.
  • Thinking p = 0.03 means a 97% chance the effect is real.
  • Treating a nonsignificant p-value as proof of no effect.
  • Equating a small p-value with a large or clinically important effect.
  • Choosing α or the one-sided test after seeing the data.
  • Ignoring multiple testing, which inflates the chance of small p-values by luck.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(103 lines)
import sys
import numpy as np
from manim import *
from blocks import *

# Parameters for the null distribution (standard normal)
MU = 0.0
SIGMA = 1.0

def pdf(x, mu=MU, sigma=SIGMA):
    """Standard normal PDF."""
    return (1.0 / (sigma * np.sqrt(2 * np.pi))) * np.exp(-0.5 * ((x - mu) / sigma) ** 2)

def cdf(x, mu=MU, sigma=SIGMA):
    """Standard normal CDF using scipy or approximation."""
    try:
        from scipy.stats import norm
        return norm.cdf(x, loc=mu, scale=sigma)
    except ImportError:
        # Simple approximation if scipy is not available
        return 0.5 * (1 + np.math.erf((x - mu) / (sigma * np.sqrt(2))))

class Main(Scene):
    def construct(self):
        # Title
        TitleCard(self, "P-value under the Null", "Probability of observing data as extreme as H1")

        # Setup Axes using CurvePlot block for robustness
        plot = CurvePlot(
            lambda x: pdf(x),
            x_range=(-3.5, 3.5, 1),
            y_range=(0, 0.5, 0.1),
            x_label="z",
            y_label="f(z)",
            color=C_MAIN,
            x_length=8.0,
            y_length=4.0
        )
        plot.move_to(ORIGIN)
        
        # Animate axes and labels
        self.play(Create(plot.axes), Write(plot.labels), run_time=1.5)

        # Plot the curve
        self.play(Create(plot.curve), run_time=2)

        # Define a specific observed statistic (e.g., z = 1.96 for p=0.05 two-tailed)
        z_obs = 1.96
        z_obs_neg = -1.96
        
        # Calculate P-value
        p_val = 2 * (1 - cdf(z_obs))
        
        # Mark the critical regions (tails)
        # Left tail
        x_left = np.linspace(-3.5, z_obs_neg, 50)
        y_left = pdf(x_left)
        points_left = [plot.axes.c2p(x, y) for x, y in zip(x_left, y_left)] + \
                      [plot.axes.c2p(z_obs_neg, 0), plot.axes.c2p(-3.5, 0)]
        poly_left = Polygon(*points_left, fill_opacity=0.4, color=C_BAD)
        
        # Right tail
        x_right = np.linspace(z_obs, 3.5, 50)
        y_right = pdf(x_right)
        points_right = [plot.axes.c2p(x, y) for x, y in zip(x_right, y_right)] + \
                       [plot.axes.c2p(3.5, 0), plot.axes.c2p(z_obs, 0)]
        poly_right = Polygon(*points_right, fill_opacity=0.4, color=C_BAD)

        # Animate the tails appearing
        self.play(FadeIn(poly_left), FadeIn(poly_right), run_time=1.5)
        
        # Add labels for the tails
        label_p = MathTex(f"P = {p_val:.3f}", font_size=32, color=C_BAD)
        label_p.next_to(poly_right, UP, buff=0.2)
        
        self.play(Write(label_p), run_time=1)
        
        # Add the observed value label
        z_label = Text(f"z = {z_obs}", font_size=24, color=C_ACCENT)
        z_label.next_to(plot.axes.c2p(z_obs, 0), DOWN, buff=0.2)
        self.play(FadeIn(z_label), run_time=0.8)

        # Explanation
        eq = MathTex(r"P = P(|Z| > z_{obs} \mid H_0 \text{ is true})", font_size=30, color=WHITE)
        eq.to_edge(UP, buff=0.8)
        self.play(Write(eq), run_time=1.5)

        StepCaption(self, "Small P-value indicates the observed data is unlikely under the null hypothesis.")
        self.wait(2)


if __name__ == "__main__" and "--selftest" in sys.argv:
    # Test PDF at mean
    assert abs(pdf(0) - 1.0 / np.sqrt(2 * np.pi)) < 1e-9
    # Test CDF at mean
    assert abs(cdf(0) - 0.5) < 1e-9
    # Test P-value for z=1.96 (approx 0.05)
    p = 2 * (1 - cdf(1.96))
    assert abs(p - 0.05) < 0.001
    # Test P-value for z=0 (should be 1.0)
    p0 = 2 * (1 - cdf(0))
    assert abs(p0 - 1.0) < 1e-9
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.