Biostatistics · Intermediate

ROC curve and AUC

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

A ROC curve plots sensitivity against 1 minus specificity as the cutoff of a diagnostic test is varied. The area under the curve (AUC) summarizes how well the test separates diseased from healthy people: 0.5 is chance, 1.0 is perfect discrimination.

Intuition

Imagine test scores for healthy and diseased patients, two overlapping bell-shaped distributions. Any cutoff you choose labels everyone above it as positive. A low cutoff catches nearly all diseased patients but also flags many healthy ones; a high cutoff does the opposite. The ROC curve traces this trade-off by sweeping the cutoff from high to low, so the point moves from the bottom-left corner (0,0) to the top-right corner (1,1). The more the two distributions are separated, the more the curve bows toward the top-left corner. AUC has a neat probabilistic meaning: it is the chance that a randomly chosen diseased patient has a higher score than a randomly chosen healthy one.

Key equations

Sensitivity=TPR=TPTP+FN
(1)

Proportion of diseased patients correctly called positive (y-axis).

1−Specificity=FPR=FPFP+TN
(2)

Proportion of healthy patients wrongly called positive (x-axis).

AUC=∫01TPRd(FPR)
(3)

Area under the ROC curve.

AUC=P(SD>SH)+12P(SD=SH)
(4)

Probability that a random diseased patient scores higher than a random healthy one, ties counted half.

J=Sens+Spec−1
(5)

Youden index, used to choose an optimal cutoff.

Derivation

  1. Choose a cutoff c; classify a patient as positive if score S ≥ c.
  2. For that c, count TP, FN, FP, TN in a 2x2 table and compute TPR and FPR.
  3. Lower c step by step: TPR and FPR can only stay the same or increase, so the curve is monotone from (0,0) to (1,1).
  4. Plot each (FPR, TPR) pair and join them; this is the empirical ROC curve.
  5. Compute the area with the trapezoidal rule between successive points.
  6. Equivalently, comparing all diseased–healthy pairs and counting the fraction in which the diseased score is higher (ties = 0.5) gives the same number, the Mann–Whitney U statistic divided by n_D·n_H.

Worked example

Problem

A test gives these (FPR, TPR) points: (0,0), (0.2,0.6), (0.5,0.9), (1,1). Estimate the AUC and the Youden-optimal point.

  1. Trapezoid 1: width 0.2, mean height (0+0.6)/2=0.3, area 0.06.
  2. Trapezoid 2: width 0.3, mean height (0.6+0.9)/2=0.75, area 0.225.
  3. Trapezoid 3: width 0.5, mean height (0.9+1)/2=0.95, area 0.475.
  4. Sum: 0.06+0.225+0.475=0.76.
  5. Youden J = TPR − FPR: 0.4 at point 2, 0.4 at point 3 (0.9−0.5), so both tie.

Answer AUC ≈ 0.76 (fair discrimination); the points (0.2,0.6) and (0.5,0.9) tie with J = 0.4.

Common mistakes

  • Thinking AUC = 0.5 means 50% accuracy in a particular sample; it means no discrimination.
  • Ignoring prevalence: ROC and AUC do not depend on prevalence, but predictive values do.
  • Choosing the cutoff closest to the corner automatically, ignoring the different clinical costs of false negatives and false positives.
  • Plotting specificity instead of 1 − specificity on the x-axis.
  • Treating AUC as calibration; a test can discriminate well yet give poorly calibrated probabilities.
  • Comparing AUCs without confidence intervals or a formal test such as DeLong.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(122 lines)
import sys
from manim import *
from blocks import *
import numpy as np

def compute_roc_trace(n_samples=200, seed=42):
    """
    Simulate a classifier with separation between classes.
    Returns a list of (FPR, TPR) points for the ROC curve.
    """
    np.random.seed(seed)
    # Class 0 (Negative): N(0, 1)
    scores_0 = np.random.normal(0, 1, n_samples)
    # Class 1 (Positive): N(1.5, 1)
    scores_1 = np.random.normal(1.5, 1, n_samples)
    
    n_pos = len(scores_1)
    n_neg = len(scores_0)
    
    # Combine and sort by score descending
    data = []
    for s in scores_0:
        data.append((s, 0))
    for s in scores_1:
        data.append((s, 1))
        
    data.sort(key=lambda x: x[0], reverse=True)
    
    points = [(0.0, 0.0)]
    tp = 0
    fp = 0
    
    for score, is_pos in data:
        if is_pos:
            tp += 1
        else:
            fp += 1
        tpr = tp / n_pos
        fpr = fp / n_neg
        points.append((fpr, tpr))
        
    return points, scores_0, scores_1

class Main(Scene):
    def construct(self):
        TitleCard(self, "ROC Curve", "Receiver Operating Characteristic")
        
        points, scores_0, scores_1 = compute_roc_trace()
        
        # Setup Axes
        axes = Axes(
            x_range=[0, 1, 0.2],
            y_range=[0, 1, 0.2],
            x_length=6,
            y_length=6,
            axis_config={"include_tip": True, "numbers_to_include": [0, 0.5, 1]},
        )
        axes.to_edge(LEFT, buff=0.5)
        axes.to_edge(DOWN, buff=0.5)
        
        x_label = MathTex("FPR", font_size=24).next_to(axes.x_axis.get_end(), DOWN)
        y_label = MathTex("TPR", font_size=24).next_to(axes.y_axis.get_end(), LEFT)
        
        self.play(Create(axes), FadeIn(x_label), FadeIn(y_label))
        
        # Draw diagonal reference line
        diag = Line(axes.c2p(0,0), axes.c2p(1,1), color=C_DIM, stroke_width=2, stroke_dasharray=[6, 6])
        self.play(Create(diag), run_time=1)
        
        # Animate the ROC curve building up
        # Create a VMobject for the curve
        roc_vmobject = VMobject()
        roc_vmobject.set_points_as_corners([axes.c2p(*p) for p in points])
        roc_vmobject.set_color(C_MAIN)
        roc_vmobject.set_stroke(width=4)
        
        # Animate creation of the curve
        self.play(Create(roc_vmobject), run_time=3)
        
        self.wait(0.5)
        
        # Calculate AUC using scipy.integrate.trapezoid or numpy.trapezoid (new name)
        # Fallback to manual sum if needed, but numpy 1.20+ has trapezoid
        try:
            auc_val = np.trapezoid([p[1] for p in points], [p[0] for p in points])
        except AttributeError:
            # Fallback for older numpy versions
            auc_val = np.trapz([p[1] for p in points], [p[0] for p in points])
        
        # Highlight AUC area
        area_points = [axes.c2p(0,0)] + [axes.c2p(*p) for p in points] + [axes.c2p(1,0)]
        area_poly = Polygon(*area_points, fill_opacity=0.2, fill_color=C_ACCENT, stroke_width=0)
        
        self.play(FadeIn(area_poly), run_time=1)
        
        # Show AUC value
        auc_text = MathTex(f"AUC = {auc_val:.2f}", font_size=36, color=C_ACCENT)
        auc_text.next_to(axes, UP, buff=0.5)
        self.play(FadeIn(auc_text))
        
        # Interpretation
        StepCaption(self, "AUC measures discriminative power: 1.0 is perfect, 0.5 is random.")
        self.wait(2)

if __name__ == "__main__" and "--selftest" in sys.argv:
    points, s0, s1 = compute_roc_trace()
    assert len(points) > 1
    assert points[0] == (0.0, 0.0)
    assert points[-1] == (1.0, 1.0)
    
    # Check monotonicity
    for i in range(1, len(points)):
        assert points[i][0] >= points[i-1][0]
        assert points[i][1] >= points[i-1][1]
        
    # Check AUC is between 0.5 and 1.0 for this setup
    try:
        auc = np.trapezoid([p[1] for p in points], [p[0] for p in points])
    except AttributeError:
        auc = np.trapz([p[1] for p in points], [p[0] for p in points])
    assert 0.5 < auc < 1.0
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.