✎ Early edition: these lessons are being reviewed.
Content
Summary
A permutation test asks whether an observed difference between groups could plausibly arise by chance if group labels were meaningless. We shuffle the labels many times, recompute the statistic each time, and see how extreme the real value is within that shuffled distribution. The proportion of shuffles at least as extreme is the p-value.
Intuition
If a treatment truly does nothing, then which patient was called 'treated' and which 'control' is arbitrary. So we can reassign labels at random and the data should look just as 'different' as the real data does. Repeating this thousands of times builds the null distribution directly from the data, with no assumption about normality. If the real difference sits far out in the tail of the shuffled differences, chance labeling rarely produces it, and we doubt the null hypothesis. The method relies on exchangeability under the null: labels can be swapped without changing the joint distribution of the data.
Key equations
(1)
Observed test statistic, here the difference in group means.
(2)
Statistic recomputed after the b-th random relabeling of the pooled data.
(3)
Two-sided Monte Carlo p-value from B random permutations; the +1 counts the observed labeling and prevents p=0.
Derivation
State H0: the outcome distribution is the same in both groups, so labels are exchangeable.
Compute T_obs from the real labeling.
Pool all n = nA + nB observations.
Randomly assign nA values to group A and nB to group B, without replacement.
Compute T* for this relabeling and store it.
Repeat B times (e.g., 10,000) to approximate the null distribution of T.
Count shuffles with |T*| at least |T_obs|; add 1 to numerator and denominator because the observed labeling is itself one valid permutation.
Divide to get ; reject if is below the chosen alpha.
Worked example
Problem
Systolic BP reduction (mmHg) in drug group A: 10, 12, 9, 15; placebo group B: 4, 6, 8, 5. Test whether the drug differs from placebo using all possible permutations.
Mean A = 46/4 = 11.5; mean B = 23/4 = 5.75; T_obs = 5.75.
Pooled data has 8 values; number of ways to choose 4 for group A is C(8,4) = 70.
The largest values are 15, 12, 10, 9 (the real A group) giving the maximum possible difference.
Next-largest difference replaces 9 with 8: A = 15,12,10,8 sum 45, mean 11.25, B sum 24, mean 6; difference 5.25 < 5.75.
So only the observed labeling reaches 5.75 in the positive direction.
By symmetry, the mirror labeling (A gets the four smallest) gives -5.75, so two-sided count = 2.
p = 2/70 ≈ 0.029.
Answer Two-sided , so at there is evidence the drug changes BP reduction.
Common mistakes
Shuffling with replacement (that is bootstrap, not permutation).
Permuting when observations are not exchangeable, e.g., paired or clustered data, without permuting within pairs or clusters.
Reporting p = 0 instead of adding 1 to numerator and denominator.
Choosing the statistic or the tail after seeing the data.
Using too few permutations, giving a noisy p-value.
Interpreting p as the probability that H0 is true.
Animation
Animation coming soon
The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.
Code
scene.py(245 lines)
import sys
import numpy as np
from manim import *
from blocks import *
def run_permutation_test(observed_diff, n_permutations=1000, seed=42):
"""
Perform a permutation test for the difference in means between two groups.
Returns the permutation distribution and the p-value.
"""
np.random.seed(seed)
# Combine groups
combined = np.concatenate([observed_diff['group1'], observed_diff['group2']])
n1 = len(observed_diff['group1'])
n2 = len(observed_diff['group2'])
perm_diffs = []
for _ in range(n_permutations):
# Shuffle
np.random.shuffle(combined)
# Split
perm_group1 = combined[:n1]
perm_group2 = combined[n1:]
# Calculate statistic
perm_diff = np.mean(perm_group1) - np.mean(perm_group2)
perm_diffs.append(perm_diff)
perm_diffs = np.array(perm_diffs)
# Two-sided p-value: proportion of permuted diffs with abs value >= observed abs diff
observed_abs = abs(observed_diff['statistic'])
p_value = np.mean(np.abs(perm_diffs) >= observed_abs)
return {
'perm_diffs': perm_diffs,
'observed_statistic': observed_diff['statistic'],
'observed_abs': observed_abs,
'p_value': p_value,
'n_permutations': n_permutations
}
# Define the data for the simulation
np.random.seed(42)
group1_data = np.random.normal(loc=5.0, scale=1.5, size=20) # Treatment
group2_data = np.random.normal(loc=4.0, scale=1.5, size=20) # Control
observed_data = {
'group1': group1_data,
'group2': group2_data,
'statistic': np.mean(group1_data) - np.mean(group2_data)
}
TRACE = run_permutation_test(observed_data, n_permutations=500, seed=42)
class Main(Scene):
def construct(self):
self.camera.background_color = "#111111"
TitleCard(self, "Permutation Test", "Non-parametric significance testing")
# --- Part 1: Show Observed Data ---
StepCaption(self, "1. Observed Difference in Means")
# Create a simple bar chart for the two groups
group_means = [np.mean(group1_data), np.mean(group2_data)]
group_labels = ["Treatment", "Control"]
# Use ArrayBars for visual representation of means
# We'll create a custom VGroup for this since ArrayBars is for single arrays
bars = VGroup()
labels = VGroup()
bar_width = 1.5
max_height = 3.0
start_x = -2.0
for i, mean in enumerate(group_means):
h = max_height * (mean / 6.0) # Normalize height
bar = Rectangle(width=bar_width, height=h, fill_color=C_MAIN if i == 0 else C_ACCENT,
fill_opacity=0.8, stroke_color=WHITE)
bar.move_to([start_x + i * (bar_width + 1.0), -1.5, 0])
bars.add(bar)
label = Text(group_labels[i], font_size=24, color=WHITE)
label.next_to(bar, DOWN, buff=0.2)
labels.add(label)
value_label = Text(f"{mean:.2f}", font_size=24, color=WHITE)
value_label.next_to(bar, UP, buff=0.1)
bars.add(value_label)
self.play(Create(bars), FadeIn(labels))
# Show observed difference
obs_diff_text = MathTex(r"\bar{x}_1 - \bar{x}_2 = " + f"{observed_data['statistic']:.2f}", font_size=32)
obs_diff_text.next_to(bars, UP, buff=0.5)
self.play(Write(obs_diff_text))
self.wait(1)
# --- Part 2: Permutation Distribution ---
StepCaption(self, "2. Permutation Distribution (Null Hypothesis)")
# Clear previous elements
self.play(FadeOut(bars), FadeOut(labels), FadeOut(obs_diff_text))
# Plot the permutation distribution
# Use CurvePlot block
perm_hist = TRACE['perm_diffs']
# Create a histogram-like plot using CurvePlot
# We need to bin the data for visualization
bins = np.linspace(min(perm_hist) - 0.5, max(perm_hist) + 0.5, 30)
hist_counts, bin_edges = np.histogram(perm_hist, bins=bins)
hist_centers = (bin_edges[:-1] + bin_edges[1:]) / 2
# Normalize to density
hist_density = hist_counts / (np.sum(hist_counts) * (bin_edges[1] - bin_edges[0]))
# Create a custom plot using Axes
axes = Axes(
x_range=[min(perm_hist) - 0.5, max(perm_hist) + 0.5, 0.5],
y_range=[0, max(hist_density) * 1.2, 0.05],
x_length=8.0,
y_length=4.0,
axis_config={"include_tip": True, "color": C_DIM}
)
axes.to_edge(LEFT, buff=0.5)
axes.to_edge(UP, buff=0.5)
x_label = Text("Difference in Means", font_size=24, color=WHITE).next_to(axes.x_axis.get_end(), RIGHT, buff=0.2)
y_label = Text("Density", font_size=24, color=WHITE).next_to(axes.y_axis.get_end(), UP, buff=0.2)
self.play(Create(axes), FadeIn(x_label), FadeIn(y_label))
# Draw the histogram bars
bar_heights = hist_density
bar_widths = bin_edges[1] - bin_edges[0]
# Calculate unit sizes from axes to ensure correct bar dimensions
# FIX: Use get_unit_size() method instead of accessing x_axis_config directly
x_unit = axes.x_axis.get_unit_size()
y_unit = axes.y_axis.get_unit_size()
for i, (center, height) in enumerate(zip(hist_centers, bar_heights)):
x_pos = axes.c2p(center, 0)[0]
y_pos = axes.c2p(0, height)[1]
bar = Rectangle(
width=bar_widths * x_unit,
height=height * y_unit,
fill_color=C_MAIN,
fill_opacity=0.6,
stroke_color=WHITE,
stroke_width=1
)
# Manually position the bar
bar.move_to([axes.c2p(center, 0)[0], axes.c2p(0, height/2)[1], 0])
self.play(Create(bar), run_time=0.05)
# Mark observed statistic
obs_x = axes.c2p(TRACE['observed_statistic'], 0)[0]
obs_line = DashedLine(
axes.c2p(TRACE['observed_statistic'], 0),
axes.c2p(TRACE['observed_statistic'], max(hist_density) * 1.2 * y_unit),
color=C_BAD,
dash_length=0.1
)
obs_label = Text("Observed", font_size=20, color=C_BAD)
obs_label.next_to(axes.c2p(TRACE['observed_statistic'], 0), DOWN, buff=0.2)
self.play(Create(obs_line), FadeIn(obs_label))
# --- Part 3: P-value ---
StepCaption(self, f"3. P-value = {TRACE['p_value']:.3f}")
# Highlight the tails
# For two-sided test, we highlight both tails beyond +/- observed_abs
# But since the distribution might not be symmetric around 0, we just highlight the area
# where |perm_diff| >= |observed_diff|
# Find the indices in the histogram that correspond to the tails
tail_indices = np.where(np.abs(hist_centers) >= TRACE['observed_abs'])[0]
# Animate the tail areas
tail_area = VGroup()
for idx in tail_indices:
center = hist_centers[idx]
height = hist_density[idx]
x_pos = axes.c2p(center, 0)[0]
y_pos = axes.c2p(0, height)[1]
tail_bar = Rectangle(
width=bar_widths * x_unit,
height=height * y_unit,
fill_color=C_BAD,
fill_opacity=0.4,
stroke_color=C_BAD,
stroke_width=1
)
tail_bar.move_to([x_pos, y_pos/2, 0])
tail_area.add(tail_bar)
self.play(FadeIn(tail_area), run_time=1.0)
# Final caption
StepCaption(self, f"Null hypothesis rejected if p < 0.05. P-value: {TRACE['p_value']:.3f}")
self.wait(2)
if __name__ == "__main__" and "--selftest" in sys.argv:
# Test that the permutation test produces valid results
assert len(TRACE['perm_diffs']) == TRACE['n_permutations']
# Test that the observed statistic is within the range of permutation diffs (roughly)
assert min(TRACE['perm_diffs']) <= TRACE['observed_statistic'] <= max(TRACE['perm_diffs'])
# Test that p-value is between 0 and 1
assert 0 <= TRACE['p_value'] <= 1
# Test that the p-value calculation is correct (recompute)
np.random.seed(42)
combined = np.concatenate([observed_data['group1'], observed_data['group2']])
n1 = len(observed_data['group1'])
n2 = len(observed_data['group2'])
n_total = n1 + n2
perm_diffs_check = []
for _ in range(500):
np.random.shuffle(combined)
perm_group1 = combined[:n1]
perm_group2 = combined[n1:]
perm_diff = np.mean(perm_group1) - np.mean(perm_group2)
perm_diffs_check.append(perm_diff)
perm_diffs_check = np.array(perm_diffs_check)
observed_abs = abs(observed_data['statistic'])
p_value_check = np.mean(np.abs(perm_diffs_check) >= observed_abs)
assert abs(TRACE['p_value'] - p_value_check) < 1e-9, f"P-value mismatch: {TRACE['p_value']} vs {p_value_check}"
print("ok")
The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).