1. Hyperparameter tuning
In this section, we will focus on hyperparameter tuning which is another method of optimizing model performance.
2. Hyperparameters
Hyperparameters are model parameters whose values are set prior to model training and control model complexity.
In parsnip, decision trees have three hyperparameters.
cost_complexity is a number that is used to penalize trees with large numbers of terminal nodes.
tree_depth controls how long the path from the root to any terminal node can be.
min_n controls the minimum data points required in a node for further splitting.
3. Default hyperparameter values
When no arguments are provided to the decision_tree() function, cost_complexity is set to 0-point-01, tree_depth to 30, and min_n to 20.
These values, however, may not be optimal for all datasets. Changing them might improve overall performance.
Hyperparameter tuning is the process of using cross validation to find the optimal set of hyperparameter values for a model.
4. Labeling hyparameters for tuning
The tune() function from the tune package is used to label model hyperparameters.
We simply set the values of each hyperparameter to tune() within decision_tree().
This creates a model object with tuning parameters and will let other methods know that they need to be optimized. When we print our dt_tune_model object, the hyperparameters appear under main arguments.
5. Creating a tuning workflow
To perform hyperparameter tuning, we first create a new workflow by updating our previous one with our new decision tree model.
We pass leads_wkfl to update_model() and replace our prior decision tree with our new one.
When we print our new workflow, we see that the hyperparameters appear in the model specification along with our feature engineering recipe.
6. Grid search
Hyperparameter tuning is accomplished using grid search, a method where a grid of hyperparameter values is generated.
For each combination, cross validation is used to estimate model performance and the best performing combination is chosen.
7. Identifying hyperparameters
The parameters function from the dials package can be used to identify the hyperparameters in a parsnip model object.
This function returns a tibble with the hyperparameters labeled by tune(), if any. This tibble is used to help generate grids for tuning.
8. Random grid
A popular method for grid search is to generate random combinations of hyperparameter values.
Since there can be an infinite number of hyperparameter combinations, choosing them at random will provide a greater chance of discovering optimal combinations. Choosing them in a systematic way may limit the range of values that are chosen which could lead to poor results.
To create a random grid, we pass the results of the parameters() function on our dt_tune_model to the grid_random() function and select the number of combinations to generate. To reproduce this grid, execute the set-dot-seed() function before grid_random().
This creates a tibble with random combinations.
9. Saving a tuning grid
To perform hyperparameter tuning on our decision tree model, we first save the results of grid_random(), where we have 5 combinations.
10. Hyperparameter tuning with cross validation
We then pass our leads_tune_wkfl to the tune_grid() function. This function also requires a cross validation object, a tuning grid, and optional custom metrics function. For our workflow, we use leads_folds, our dt_grid, and our custom leads_metrics function.
tune_grid() returns a tibble of tuning results. The dot-metrics column is a list column with the results for each fold.
11. Exploring tuning results
The collect_metrics() functions can be used on a tuning object to collect results. By default, the average performance metric is returned for each combination of hyperparameter values and performance metric.
Since our decision tree has 5 random hyperparameter combinations and we are tracking 3 metrics, we get 15 total rows in the output. Since we have 10 folds in our cross validation object, each row is the average of the performance across the 10 folds.
12. Let's get tuning!
Let's practice tuning decision trees!