Biostatistics · Intermediate

Tests of Significance

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Biostatistics provides the methodology to distinguish true effects from natural variation using statistical significance, typically defined by a P-value less than 0.05. This threshold corresponds to a 95% confidence level, where results falling outside two standard deviations of the mean in a normal distribution are considered unlikely to be due to chance alone.

Intuition

Natural variation means that observed differences in data often arise from random chance rather than the intervention itself, such as a patient recovering due to spontaneous healing rather than a drug. To determine if a result is real, researchers compare their findings against a baseline, often using a control group, to see if the difference exceeds the limits of this random fluctuation. The concept of the normal distribution helps visualize this, where most data clusters around the mean, and only extreme outliers in the 'tails' are deemed statistically significant.

Derivation

  1. Natural variation causes observed results to differ from expected probabilities due to chance factors like placebo effects or biological variability.
  2. To quantify this, researchers use the normal distribution, where approximately 95% of observations fall within two standard deviations (SD) of the mean.
  3. The remaining 5% of observations reside in the tails of the distribution, representing rare events.
  4. A P-value less than 0.05 indicates that the probability of the observed result occurring by chance is less than 5%.
  5. Therefore, if a result falls outside the two-SD range (in the tails), it is considered statistically significant, implying a confidence level greater than 95% that the effect is real.

Worked example

Problem

Drug B shows a 50% success rate compared to Drug A’s 38.09%. A statistical test yields a P value of 0.74. Is this difference statistically significant?

  1. Identify the P value from the test: 0.74.
  2. Compare the P value to the conventional significance threshold of 0.05.
  3. Since 0.74 is greater than 0.05, the probability that the difference occurred by chance is high.
  4. Conclude that the observed 12% gap is not statistically significant and likely due to natural variation.

Answer No, the difference is not statistically significant because the P value (0.74) exceeds the 0.05 threshold.

Common mistakes

  • Assuming that a large observed difference between groups is automatically significant without statistical validation.
  • Dismissing a difference as insignificant solely based on visual inspection, ignoring the need for formal testing.
  • Interpreting the term 'normal distribution' as implying medical health status rather than the shape of the statistical curve.
  • Believing that a single observation, such as nine heads in ten coin tosses, proves a change in underlying probability rather than natural variation.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.