AI, machine learning and deep learning · Introductory

k-fold cross-validation

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

k-fold cross-validation estimates how well a model will perform on unseen data by splitting the dataset into k equal parts, training on k-1 parts and testing on the remaining part, and rotating until every part has served as the test set. The k scores are averaged to give a more stable performance estimate than a single train/test split.

Intuition

A single train/test split depends on luck: an easy or hard test set can make a model look better or worse than it really is. Cross-validation reduces this luck by giving every observation a turn in the test set. Think of a student who sits k different mock exams, each made of material they did not study from, and whose average mark is a fairer guide to the real exam. Each fold is used for testing exactly once and for training k-1 times. Importantly, the model is retrained from scratch in every round, so no information leaks from a test fold into training.

Key equations

CVk=1k∑i=1kEi
(1)

The cross-validation estimate is the mean of the k fold errors (or scores) E_i.

ntest≈nk,ntrain≈(k−1)nk
(2)

Each fold holds about n/k samples; the training set uses the rest.

SE=sk
(3)

A rough standard error of the mean score, where s is the standard deviation of the fold scores (approximate, since folds are not independent).

Derivation

  1. Start with a dataset of n labelled samples and shuffle it randomly (stratify by class if classes are imbalanced).
  2. Partition it into k folds of about n/k samples each.
  3. For i = 1 to k: hold out fold i as the test set, train a new model on the other k-1 folds, and compute the error E_i on fold i.
  4. Because every sample is tested exactly once, all n samples contribute to the evaluation.
  5. Average the errors: CV_k = (1/k) times the sum of E_i. Report the spread too.
  6. Choose between models or hyperparameters by the lowest CV_k, then retrain the final model on all the data.

Worked example

Problem

A classifier is evaluated with 5-fold cross-validation on 100 samples. The fold accuracies are 80%, 85%, 90%, 75%, 80%. Find the mean accuracy, the size of each training set and the standard deviation of the fold scores.

  1. Each fold has 100/5 = 20 test samples, so each training set has 80 samples.
  2. Mean = (80+85+90+75+80)/5 = 410/5 = 82%.
  3. Deviations from the mean: -2, 3, 8, -7, -2; squares: 4, 9, 64, 49, 4; sum = 130.
  4. Sample variance = 130/(5-1) = 32.5, so s ≈ 5.7 percentage points.

Answer Mean accuracy 82% with standard deviation about 5.7 points; 80 training and 20 test samples per round.

Common mistakes

  • Preprocessing (scaling, feature selection, imputation) on the whole dataset before splitting, which leaks test information; do it inside each training fold.
  • Not shuffling or stratifying, so folds have unrepresentative class balance or ordered data.
  • Reusing a trained model across folds instead of retraining from scratch.
  • Treating the CV score as the final unbiased test result after heavy tuning; keep a separate untouched test set.
  • Using ordinary k-fold on time series or grouped data (e.g. several samples per patient), which leaks information; use time-based or group-wise splits.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

scene.py(77 lines)
import sys
from manim import *
from blocks import *


def kfold_trace(n, k):
    """Split n samples into k contiguous folds; record each round's train/test indices."""
    sizes = [n // k + (1 if i < n % k else 0) for i in range(k)]
    folds, start = [], 0
    for s in sizes:
        folds.append(list(range(start, start + s)))
        start += s
    rounds = []
    for f in range(k):
        test = folds[f]
        train = [i for g in range(k) if g != f for i in folds[g]]
        rounds.append({"fold": f, "test": test, "train": train})
    return folds, rounds


N, K = 10, 5
FOLDS, ROUNDS = kfold_trace(N, K)


class Main(Scene):
    def construct(self):
        self.camera.background_color = "#14161c"
        TitleCard(self, "k-fold cross-validation", "k = 5, 10 samples")

        # grid of rounds: rows = rounds, columns = samples
        cs = 0.6
        cells = {}
        grid = VGroup()
        for r in range(K):
            for i in range(N):
                sq = Square(side_length=cs, stroke_width=2, color=C_DIM,
                            fill_color=C_MAIN, fill_opacity=0.5)
                sq.move_to([(i - (N - 1) / 2) * cs, -r * cs, 0])
                cells[(r, i)] = sq
                grid.add(sq)
        grid.move_to(ORIGIN + 0.1 * UP)
        row_labels = VGroup()
        for r in range(K):
            t = Text(f"Round {r + 1}", font_size=24)
            t.next_to(cells[(r, 0)], LEFT, buff=0.3)
            row_labels.add(t)
        self.play(FadeIn(grid), FadeIn(row_labels))
        legend = VGroup(
            Square(0.25, fill_color=C_MAIN, fill_opacity=0.5, color=C_MAIN),
            Text("train", font_size=24),
            Square(0.25, fill_color=C_ACCENT, fill_opacity=0.9, color=C_ACCENT),
            Text("test", font_size=24),
        ).arrange(RIGHT, buff=0.2)
        legend.next_to(grid, UP, buff=0.3).shift(0.3 * DOWN * 0)
        legend.to_edge(RIGHT, buff=0.5).shift(UP * 1.5)
        self.play(FadeIn(legend))

        # animate recorded rounds
        StepCaption(self, "Each round holds out a different fold for testing")
        for rd in ROUNDS:
            r = rd["fold"]
            self.play(*[cells[(r, i)].animate.set_fill(C_ACCENT, opacity=0.9)
                        for i in rd["test"]], run_time=0.7)
        StepCaption(self, "Every sample is tested exactly once; average the k scores")


if __name__ == "__main__" and "--selftest" in sys.argv:
    folds, rounds = kfold_trace(10, 5)
    assert len(rounds) == 5
    for rd in rounds:
        assert set(rd["train"]).isdisjoint(rd["test"])
        assert sorted(rd["train"] + rd["test"]) == list(range(10))
    all_test = sorted(i for rd in rounds for i in rd["test"])
    assert all_test == list(range(10))
    f2, r2 = kfold_trace(11, 3)
    assert [len(f) for f in f2] == [4, 4, 3]
    print("ok")

The Manim scene code that draws this lesson's animation in the Signals AI app (Python, Manim Community).

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.