AI, machine learning and deep learning · Intermediate

Learning with Imbalanced Data

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Imbalanced data arises when the underlying distribution is skewed, causing standard accuracy-optimizing algorithms to ignore the minority class. To address this, we redefine the learning objective using metrics like F-score or AUC and apply heuristics such as subsampling or weighting to balance class influence. The α-weighted classification framework explicitly adjusts the cost of misclassification to prioritize the minority class.

Intuition

Imagine a fraud detection system where 99.9% of transactions are legitimate. A naive algorithm can achieve 99.9% accuracy by simply predicting 'legitimate' for every single transaction. While this looks good on paper, it is useless because it fails to detect any fraud. The problem isn't the data itself, but the distribution from which it is drawn. To fix this, we must stop optimizing for overall accuracy and start optimizing for metrics that care about the minority class, like F-score or AUC. We can do this by either throwing away some majority class examples (subsampling) or by telling the algorithm that missing a fraud case is α times more costly than a false alarm (weighting). This shifts the model's focus from being 'generally correct' to being 'correct where it matters most'.

Key equations

E(x,y)∼D[f(x)≠y]
(1)

The expected error for standard binary classification, where f is the classifier, x is the input, y is the label, and D is the distribution.

E(x,y)∼D[αy=1⋅𝕀(f(x)≠y)]
(2)

The expected error for α-weighted binary classification, where α is the importance weight for positive examples (y=1) and 1 is the weight for negative examples (y=-1).

Derivation

  1. Standard binary classification minimizes the expected error E_{(x,y) ~ D} [f(x) != y], which is simply the probability of misclassification.
  2. In imbalanced datasets, the majority class dominates this error term, allowing the model to achieve low error by always predicting the majority class.
  3. To prioritize the minority class, we introduce an importance weight α for positive examples (y=1).
  4. The new objective becomes minimizing E_{(x,y) ~ D} [α_{y=1} · I(f(x) != y)], where the cost of mispredicting a positive instance is α and a negative instance is 1.
  5. This α-weighted problem can be solved by subsampling: retaining all positive examples and a 1/α fraction of negative examples, effectively mapping the weighted problem to an unweighted one.
  6. If α < 1, labels can be swapped and 1/α used to maintain the same optimization logic.

Worked example

Problem

Consider a fraud detection dataset where 0.1% of transactions are fraudulent. You want to use subsampling to create a balanced dataset (50% positive, 50% negative) to train a vanilla binary classifier. How should you apply the SubsampleMap algorithm?

  1. Identify the minority class (fraud) and majority class (legitimate).
  2. The goal is to retain all positive examples (fraudulent transactions).
  3. Calculate the fraction of negative examples to retain. Since we want a 50/50 balance and the original positive ratio is very small, we need to reduce the negative class significantly.
  4. According to the SubsampleMap algorithm, we retain all positive examples and a 1/α fraction of negative examples. In this specific heuristic context for creating a 50/50 split from a highly skewed distribution, we effectively treat the problem as if we are balancing the weights by discarding the majority.
  5. By discarding most negative examples, we force the vanilla classifier to learn from a balanced distribution, preventing it from ignoring the minority class.

Answer Retain all fraudulent transactions and subsample the legitimate transactions such that the resulting dataset is 50% fraudulent and 50% legitimate. This transforms the weighted problem into an unweighted one solvable by a standard classifier.

Common mistakes

  • Thinking that the data itself is imbalanced rather than the underlying distribution from which it is drawn.
  • Assuming that high accuracy is a good metric for imbalanced datasets, when it often masks the model's failure to predict the minority class.
  • Believing that throwing out data (subsampling) is always a bad idea, whereas it can make learning more efficient and is theoretically optimal for certain reductions.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.