Content
Summary
Data cleaning and preprocessing transform raw information into accurate, consistent structures through six sequential steps: integration, cleaning, transformation, reduction, discretization, and sampling. This process addresses missing data mechanisms (MCAR, MAR, MNAR), manages outliers via robust methods like median replacement, and applies normalization to scale values between zero and one.
Intuition
Think of data cleaning as preparing a messy room for a guest. You first organize the sources (integration), then remove trash and fix broken items (cleaning), arrange furniture for better flow (transformation), and decide what is truly necessary (reduction). Missing data is like a gap in a photo; if it's random, it's MCAR, but if it's related to who is in the photo, it's MAR. Outliers are like a giant elephant in a small room; you can't just ignore it, but you might replace it with the average size of the other animals to keep the room's proportions realistic. Finally, normalization ensures all items are measured on the same scale, so a small toy doesn't look bigger than a large car just because of how you measured them.
Key equations
The formula for statistical normalization, scaling a value x to a range between 0 and 1 based on the minimum and maximum values in the dataset.
Derivation
- Data preprocessing begins with integration, merging multiple sources into a unified dataset.
- Cleaning assesses and corrects inconsistencies, handling missing values via deletion or imputation.
- Transformation prepares data for analysis, often addressing statistical assumptions like normality.
- Reduction selects the most appropriate features to maintain data integrity.
- Discretization groups continuous data into categories, and sampling selects subsets.
- Normalization scales data to [0,1] using the min-max formula, requiring outlier removal first to prevent skewing min/max values.
Worked example
A retail company analyzes sales data for eight products. The highest recorded sale is USD 55,000 and the lowest is USD 15,000. Calculate the normalized value for Jewelry, which has sales of USD 50,000.
- Identify the normalization formula: .
- Substitute the given values: , , and .
- Calculate the numerator: .
- Calculate the denominator: .
- Divide the numerator by the denominator: .
- Round to two decimal places as per the source example: 0.88.
Answer 0.88
Common mistakes
- Applying normalization before removing outliers, which skews the min and max values.
- Treating all missing data as MCAR, ignoring that missingness may depend on observed (MAR) or unobserved (MNAR) variables.
- Deleting outliers without considering if they represent genuine extremes or valuable information.
- Using the mean instead of the median when robust methods are needed to mitigate outlier influence.