Content
Summary
Data transformation and discretization are preprocessing steps that consolidate raw data into forms suitable for efficient mining and easier interpretation. These techniques include normalization to scale attributes, discretization to convert continuous values into intervals or concepts, and the generation of concept hierarchies for multi-level abstraction.
Intuition
Imagine you are trying to find patterns in a massive, messy dataset. Raw data often comes in different units (like height in meters vs. inches) or has too much noise and detail to be useful. Transformation strategies like normalization standardize these scales so no single attribute dominates the analysis. Discretization goes a step further by grouping continuous numbers into meaningful buckets, such as turning specific ages into 'youth' or 'adult'. This not only makes the data easier to understand but also reduces the complexity, allowing algorithms to mine patterns more efficiently across different levels of abstraction, from individual details to broad summaries.
Derivation
- Raw data often contains noise, varying units, or excessive detail, making mining inefficient.
- Normalization scales attributes to a specific range (e.g., 0.0 to 1.0) to handle different measurement units.
- Discretization replaces continuous numeric values with interval or conceptual labels (e.g., 0–10 or youth/adult).
- Discretization can be top-down (splitting intervals recursively) or bottom-up (merging values recursively).
- Discretization and concept hierarchy generation act as data reduction techniques, simplifying data for mining.
- Concept hierarchies allow mining at multiple abstraction levels, from base cuboids (lowest level) to apex cuboids (highest level).
Worked example
You have a dataset of daily sales figures that is noisy and too granular. How would you apply transformation and discretization to make it suitable for mining monthly trends?
- First, apply smoothing to remove noise from the daily sales data.
- Next, use aggregation to summarize the data, such as computing monthly totals from the daily sales.
- Then, discretize the monthly totals into intervals (e.g., low, medium, high sales) to reduce the number of distinct values.
- Finally, organize these intervals into a concept hierarchy to allow analysis at different levels of abstraction.
Answer By smoothing noise, aggregating daily sales to monthly totals, discretizing those totals into intervals, and organizing them into a concept hierarchy, the data becomes simpler, reduced in size, and easier to mine for patterns.
Common mistakes
- Confusing discretization with general data cleaning; while smoothing is a form of data cleaning, discretization is specifically about converting continuous values to intervals or concepts.
- Misunderstanding the direction of discretization: top-down starts with split points and splits intervals, whereas bottom-up starts with all values and merges them.
- Overlooking that discretization is also a data reduction technique, not just a transformation for readability.
- Assuming all concept hierarchies are explicit; many for nominal data are implicit in the database schema.