1. Tuning an RF's Hyperparameters
Let's now turn to a case where we tune the hyperparameters of Random Forests which is an ensemble method.
2. Random Forests Hyperparameters
In addition to the hyperparameters of the CARTs forming random forests, the ensemble itself is characterized by other hyperparameters such as the number of estimators, whether it uses bootstraping or not and so on.
3. Tuning is expensive
As a note, hyperparameter tuning is computationally expensive and may sometimes lead only to very slight improvement of a model's performance.
For this reason, it is desired to weigh the impact of tuning on the pipeline of your data analysis project as a whole in order to understand if it is worth pursuing.
4. Inspecting RF Hyperparameters in sklearn
To inspect the hyperparameters of a RandomForestRegressor, first, import RandomForestRegressor from sklearn.ensemble and then instantiate a RandomForestRegressor rf as shown here.
5. Inspecting RF Hyperparameters in sklearn
The hyperparameters of rf along with their default values can be accessed by calling rf's dot-get_params() method.
In the following, we'll be optimizing n_estimators, max_depth, min_samples_leaf and max_features.
You can learn more about these hyperparameters by consulting scikit-learn's documentation.
6. GridSearchCV in sklearn (auto dataset)
We'll perform grid-search cross-validation on the auto-dataset which is already loaded and split into 80%-train and 20%-test.
First import mean_squared_error as MSE from sklearn.metrics and GridSearchCV from sklearn.model_selection.
Then, define a dictionary called params_rf containing the grid of hyperparameters.
Finally, instantiate a GridSearchCV object called grid_rf and pass the parameters rf as estimator, params_rf as param_grid. Also set cv to 3 to perform 3-fold cross-validation. In addition, set scoring to neg_mean_squared_error in order to use negative mean squared error as a metric.
Note that the parameter verbose controls verbosity; the higher its value, the more messages are printed during fitting.
7. Searching for the best hyperparameters
You can now fit grid_rf to the training set as shown here.
The output shows messages related to grid fitting as well as the obtained optimal model.
8. Extracting the best hyperparameters
You can extract rf's best hyperparameters by getting the attribute best_params_ from grid_rf.
The results are shown here.
9. Evaluating the best model performance
You can also extract the best model from rf. This enables you to predict the test set labels and evaluate the test-set RMSE.
The output shows a result of 3-dot-89. If you would have trained an untuned model, the RMSE would be 3-dot-98.
10. Let's practice!
Now let's try some examples.