1. Random Forests
You will now learn about another ensemble learning method known as Random Forests.
2. Bagging
Recall that in bagging the base estimator could be any model including a decision tree, logistic regression or even a neural network.
Each estimator is trained on a distinct bootstrap sample drawn from the training set using all available features.
3. Further Diversity with Random Forests
Random Forests is an ensemble method that uses a decision tree as a base estimator.
In Random Forests, each estimator is trained on a different bootstrap sample having the same size as the training set.
Random forests introduces further randomization than bagging when training each of the base estimators.
When each tree is trained, only d features can be sampled at each node without replacement, where d is a number smaller than the total number of features.
4. Random Forests: Training
The diagram here shows the training procedure for random forests. Notice how each tree forming the ensemble is trained on a different bootstrap sample from the training set.
In addition, when a tree is trained, at each node, only d features are sampled from all features without replacement.
The node is then split using the sampled feature that maximizes information gain.
In scikit-learn d defaults to the square-root of the number of features. For example, if there are 100 features, only 10 features are sampled at each node.
5. Random Forests: Prediction
Once trained, predictions can be made on new instances. When a new instance is fed to the different base estimators, each of them outputs a prediction.
The predictions are then collected by the random forests meta-classifier and a final prediction is made depending on the nature of the problem.
6. Random Forests: Classification & Regression
For classification, the final prediction is made by majority voting. The corresponding scikit-learn class is RandomForestClassifier.
For regression, the final prediction is the average of all the labels predicted by the base estimators. The corresponding scikit-learn class is RandomForestRegressor.
In general, Random Forests achieves a lower variance than individual trees.
7. Random Forests Regressor in sklearn (auto dataset)
Alright, now it's time to put all this into practice. Here, you'll train a random forests regressor to the auto-dataset which you were introduced to in previous chapters. Note that the dataset is already loaded.
After importing RandomForestRegressor, train_test_split and mean_squared_error as MSE, split the dataset into 70%-train and 30%-test as shown here.
8. Random Forests Regressor in sklearn (auto dataset)
Then instantiate a RandomForestRegressor consisting of 400 regression trees. This can be done by setting n_estimators to 400. In addition, set min_samples_leaf to 0-dot-12 so that each leaf contains at least 12% of the data used in training.
You can now fit rf to the training set and predict the test set labels.
Finally, print the test set RMSE.
The result shows that rf achieves a test set RMSE of 3-dot-98; this error is smaller than that achieved by a single regression tree which is 4-dot-43.
9. Feature Importance
When a tree based method is trained, the predictive power of a feature or its importance can be assessed.
In scikit-learn, feature importance is assessed by measuring how much the tree nodes use a particular feature to reduce impurity.
Note that the importance of a feature is expressed as a percentage indicating the weight of that feature in training and prediction.
Once you train a tree-based model in scikit-learn, the features importances can be accessed by extracting the feature_importance_ attribute from the model.
10. Feature Importance in sklearn
To visualize the importance of features as assessed by rf, you can create a pandas series of the features importances as shown here and then sort this series and make a horiztonal-barplot.
11. Feature Importance in sklearn
The results show that, according to rf, displ, size, weight and hp are the most predictive features.
12. Let's practice!
Now let's try some examples.