AI, machine learning and deep learning · Introductory

Handling Missing Values

Written by SignalsLab

Early edition: these lessons are being reviewed.

Content

Summary

Missing values, denoted by NumPy's NaN in Pandas, are a critical data wrangling issue that can lead to incorrect interpretations if untreated. Detection relies on methods like isnull and notnull, while handling involves specifying custom missing indicators during import or replacing specific values with NaN after importing NumPy.

Intuition

Think of missing values as silent gaps in your dataset that distort the picture. Pandas uses NumPy's NaN to mark these gaps, but since NaN isn't native to Pandas, you must explicitly import NumPy to use it. You can spot these gaps by checking if a column's count in describe() is less than the total rows, or by using boolean methods. It is crucial to distinguish between truly missing data (like a refused survey answer) and technical artifacts (like extraction errors), as treating them all the same can skew your statistical summaries and model performance.

Key equations

pd.isnull(data[′body′]).values.ravel().sum()
(1)

Calculates the total number of missing values in the 'body' column by converting the boolean series to binary (1 for missing, 0 for not) and summing them.

dataframe[′Sex′]=dataframe[′Sex′].replace(′male′,np.nan)
(2)

Replaces all occurrences of the string 'male' in the 'Sex' column with NaN, requiring the NumPy library import.

Derivation

  1. Missing values are ubiquitous in data wrangling and can lead to incorrect interpretations if left untreated.
  2. Pandas utilizes NumPy's NaN to denote missing values, but NumPy must be imported to use NaN effectively.
  3. Detection is performed using isnull and notnull methods, which return boolean series indicating missing status.
  4. A column's count in describe() being less than the total rows indicates the presence of missing values.
  5. Specific values like NONE or -999 can be specified as missing using the na_values parameter in read_csv.
  6. Values can be replaced with NaN using the replace method, provided NumPy is imported.

Worked example

Problem

You have a dataset where the 'age' column has 1046 entries out of 1310 rows, and the 'body' column has only 121 entries. How do you confirm the number of missing values in 'body' and replace any string 'male' in a 'Sex' column with NaN?

  1. Check the 'body' column count: 121 entries exist out of 1310, meaning 1189 are missing.
  2. Calculate the exact count of missing values in 'body' using pd.isnull(data['body']).values.ravel().sum().
  3. Import NumPy to access np.nan.
  4. Replace 'male' strings in the 'Sex' column using dataframe['Sex'].replace('male', np.nan).

Answer There are 1189 missing values in 'body'. The code to count them is pd.isnull(data['body']).values.ravel().sum(). The code to replace 'male' with NaN is dataframe['Sex'] = dataframe['Sex'].replace('male', np.nan).

Common mistakes

  • Attempting to use NaN without importing NumPy results in a NameError because NaN is not fully implemented natively in pandas.
  • Confusing 'extra' classes in categorical data (like '*' in PClass) with missing values; while both need handling, they are distinct issues.
  • Assuming that all values shown as missing due to extraction incompatibilities are actually missing; some can be corrected by optimizing the extraction process.

Animation

Animation coming soon

The animated version of this lesson is being prepared for the web. In the Signals AI app, lessons are drawn step by step.

Code

This is a text lesson: it has no animation code yet.

ccBYNC© 2026 SignalsLab. Lesson text shared as is under CC BY-NC 4.0.