AI, machine learning and deep learning · Introductory

Use Simulations, but Trust Real Data

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Understanding data analysis requires explaining its mechanics, interpreting results, assessing parameter effects, and detecting errors, rather than necessarily possessing deep mathematical mastery. Simulated data serves as a critical tool for this understanding by providing known ground truth, but researchers must ultimately validate methods on real data due to potential mismatches in characteristics.

Intuition

Think of understanding an analysis not as memorizing a formula, but as knowing how to drive a car: you need to know how it works, how to read the dashboard, how the pedals affect speed, and how to spot if the engine is making a strange noise. While writing code from scratch is the ideal deep understanding, a realistic standard is being able to explain these four aspects clearly. Simulated data acts like a driving simulator where you know exactly what the road looks like, helping you test the car's limits safely. However, because the simulator isn't the real road, you must eventually drive on actual streets to ensure the car performs well in unpredictable, real-world conditions.

Derivation

  1. Step 1: Define the spectrum of understanding, ranging from deep mathematical mastery (programming from scratch) to a realistic standard of explaining mechanics, interpretation, parameters, and error detection.
  2. Step 2: Identify programming as the essential skill that bridges theory and application by allowing systematic testing of parameters and inputs.
  3. Step 3: Establish simulated data as a primary evaluation tool because it provides known ground truth, facilitating method development and behavior verification.
  4. Step 4: Recognize the limitation that simulated data often lacks real-world characteristics, creating a risk that methods fine-tuned on simulations may fail on real data.
  5. Step 5: Conclude that a two-step evaluation strategy is required: first using simulations for known outcomes, then applying to real data for final validation and careful interpretation.

Worked example

Problem

A researcher develops a new clustering algorithm. It achieves perfect separation on a simulated dataset where clusters are perfectly spherical and evenly spaced. The researcher publishes the results without testing on real data. What is the flaw in this approach?

  1. The researcher relied solely on simulated data where the ground truth was known and conditions were ideal.
  2. Simulated data often lacks the noise, irregular shapes, and complex distributions found in real data.
  3. The algorithm may have been fine-tuned to capture simulated dynamics (perfect spheres) that do not exist in reality.
  4. Without testing on real data, the researcher cannot verify if the method produces sensible results or detects errors in practical scenarios.

Answer The flaw is assuming success in simulated data guarantees success in real data. The method likely overfitted to the artificial simplicity of the simulation and may fail or produce nonsensical results when applied to the messy, unstructured nature of real-world empirical data.

Common mistakes

  • Assuming that success in simulated data guarantees success in real data.
  • Believing that one must have a deep mathematical understanding of every analysis to use it effectively.
  • Fine-tuning methods based on simulated data without verifying if the simulated dynamics exist in real data.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.