1. Out Of Bag Evaluation
You will now learn about Out-of-bag evaluation.
2. Bagging
Recall that in bagging, some instances may be sampled several times for one model.
On the other hand, other instance may not be sampled at all.
3. Out Of Bag (OOB) instances
On average, for each model, 63% of the training instances are sampled.
The remaining 37% that are not sampled constitute what is known as the Out-of-bag or OOB instances.
Since OOB instances are not seen by a model during training, these can be used to estimate the performance of the ensemble without the need for cross-validation. This technique is known as OOB-evaluation.
4. OOB Evaluation
To understand OOB-evaluation more concretely, take a look at this diagram.
Here, for each model, the bootstrap instances are shown in blue while the OOB-instances are shown in red.
Each of the N models constituting the ensemble is then trained on its corresponding bootstrap samples and evaluated on the OOB instances.
This leads to the obtainment of N OOB scores labeled OOB1 to OOBN. The OOB-score of the bagging ensemble is evaluated as the average of these N OOB scores as shown by the formula on top.
5. OOB Evaluation in sklearn (Breast Cancer Dataset)
Alright! Now it's time to see OOB-evaluation in action. Again, we'll be classifying cancerous cells as malignant or benign from the breast cancer dataset which is already loaded.
After importing BaggingClassifier, DecisionTreeClassifier, accuracy_score and train_test_split, split the dataset in a stratified way into 70%-train and 30%-test by setting the parameter stratify to y.
6. OOB Evaluation in sklearn (Breast Cancer Dataset)
Now, first instantiate a classification tree dt with a maximum-depth of 4 and a minimum percentage of samples per leaf equal to 16%.
Then instantiate a BaggingClassifier called bc that consists of 300 classification trees. This can be done by setting the parameters n_estimators to 300 and base_estimator to dt.
Importantly, set the parameter oob_score to True in order to evaluate the OOB-accuracy of bc after training.
Note that in scikit-learn, the OOB-score corresponds to the accuracy for classifiers and the r-squared score for regressors.
Now fit bc to the training set and predict the test set labels.
7. OOB Evaluation in sklearn (Breast Cancer Dataset)
Assign the test set accuracy to test_accuracy.
Finally, evaluate the OOB-accuracy of bc by extracting the attribute oob_score_ from the trained instance; assign the result to oob_accuracy and print out the results.
The test-set accuracy is about 93.6% and the OOB-accuracy is about 92.5%.
The two obtained accuracies are pretty close though not exactly equal. These results highlight how OOB-evaluation can be an efficient technique to obtain a performance estimate of a bagged-ensemble on unseen data without performing cross-validation.
8. Let's practice!
Now let's try some examples.