Content
Summary
k-fold cross-validation estimates how well a model will perform on unseen data by splitting the dataset into k equal parts, training on k-1 parts and testing on the remaining part, and rotating until every part has served as the test set. The k scores are averaged to give a more stable performance estimate than a single train/test split.
Intuition
A single train/test split depends on luck: an easy or hard test set can make a model look better or worse than it really is. Cross-validation reduces this luck by giving every observation a turn in the test set. Think of a student who sits k different mock exams, each made of material they did not study from, and whose average mark is a fairer guide to the real exam. Each fold is used for testing exactly once and for training k-1 times. Importantly, the model is retrained from scratch in every round, so no information leaks from a test fold into training.
Key equations
The cross-validation estimate is the mean of the k fold errors (or scores) E_i.
Each fold holds about n/k samples; the training set uses the rest.
A rough standard error of the mean score, where s is the standard deviation of the fold scores (approximate, since folds are not independent).
Derivation
- Start with a dataset of n labelled samples and shuffle it randomly (stratify by class if classes are imbalanced).
- Partition it into k folds of about n/k samples each.
- For i = 1 to k: hold out fold i as the test set, train a new model on the other k-1 folds, and compute the error E_i on fold i.
- Because every sample is tested exactly once, all n samples contribute to the evaluation.
- Average the errors: CV_k = (1/k) times the sum of E_i. Report the spread too.
- Choose between models or hyperparameters by the lowest CV_k, then retrain the final model on all the data.
Worked example
A classifier is evaluated with 5-fold cross-validation on 100 samples. The fold accuracies are 80%, 85%, 90%, 75%, 80%. Find the mean accuracy, the size of each training set and the standard deviation of the fold scores.
- Each fold has 100/5 = 20 test samples, so each training set has 80 samples.
- Mean = (80+85+90+75+80)/5 = 410/5 = 82%.
- Deviations from the mean: -2, 3, 8, -7, -2; squares: 4, 9, 64, 49, 4; sum = 130.
- Sample variance = 130/(5-1) = 32.5, so s ≈ 5.7 percentage points.
Answer Mean accuracy 82% with standard deviation about 5.7 points; 80 training and 20 test samples per round.
Common mistakes
- Preprocessing (scaling, feature selection, imputation) on the whole dataset before splitting, which leaks test information; do it inside each training fold.
- Not shuffling or stratifying, so folds have unrepresentative class balance or ordered data.
- Reusing a trained model across folds instead of retraining from scratch.
- Treating the CV score as the final unbiased test result after heavy tuning; keep a separate untouched test set.
- Using ordinary k-fold on time series or grouped data (e.g. several samples per patient), which leaks information; use time-based or group-wise splits.