Content
Summary
A Random Forest Classifier is an ensemble method that constructs multiple randomized decision trees using bootstrapped samples and random feature subsets to mitigate overfitting. The final prediction is determined by majority voting among the individual trees, offering parallelizable training and robust generalization. This approach reduces variance compared to single decision trees without requiring feature scaling.
Intuition
Think of a Random Forest as a committee of experts where each expert (decision tree) makes an independent guess based on a slightly different view of the data. By using bootstrapped samples (sampling with replacement) and considering only random subsets of features at each split, we introduce randomness that prevents any single tree from memorizing the noise in the training data. This diversity ensures that while individual trees might be biased or overfit, their collective agreement (majority vote) provides a stable and accurate prediction. The process is parallelizable because each tree is grown independently, allowing the model to scale efficiently across multiple processors.
Key equations
The final classifier output is the sign of the sum of predictions from K individual trees, representing the voting mechanism.
Derivation
- The algorithm initializes a forest of K decision trees.
- For each tree k, a bootstrapped sample of observations is drawn from the training data.
- During the growth of each tree, at every node split, only a random subset of features is considered.
- Each tree is grown independently to its maximum depth (or constrained by max_depth).
- For a new input , each tree f^{(k)} produces a prediction.
- The final prediction is determined by aggregating these predictions via majority vote (sign of the sum).
Worked example
You have a dataset with 1000 instances and 50 features. You need to train a Random Forest Classifier with 100 trees. How does the training process handle feature selection and parallelization?
- The model will create 100 independent decision trees.
- For each tree, a bootstrapped sample of the 1000 instances is selected.
- At each node split within a tree, the algorithm considers only a random subset of the 50 features, not all of them.
- Since each tree is built independently, the training of these 100 trees can be parallelized across multiple processors.
- No feature scaling is required before training.
Answer The Random Forest builds 100 trees in parallel, each using a random subset of features and bootstrapped data, eliminating the need for feature scaling and reducing overfitting through ensemble voting.
Common mistakes
- Assuming that every child node in a decision tree must have lower Gini impurity than its parent; a child can have higher impurity if the weighted average of children is lower.
- Believing that presorting data always speeds up training; it only helps for small datasets (< few thousand instances) and slows down large ones.
- Thinking that boosting ensembles can be parallelized like bagging or random forests; boosting is sequential.
- Assuming that scaling features improves decision tree performance; it is unnecessary and wastes time.
- Confusing Random Forests with Extra-Trees; Extra-Trees use random thresholds, while Random Forests search for best thresholds.