1. Selecting the best model
In the previous section, we performed hyperparameter tuning and obtained a tibble of results. The next step is to select the best performing hyperparameter value combinations and finalize our workflow.
2. Detailed tuning results
By default, the collect_metrics() function provides summarized results from hyperparameter tuning.
Just like in the resampling case with fit_resamples(), passing summarize equals FALSE into collect_metrics() will produce a tibble of detailed results. The detailed results for our dt_tuning object includes 150 rows. This is because we have 10 folds, 3 metrics, and 5 random hyperparameter combinations that were evaluated.
3. Exploring tuning results
Since collect_metrics() returns a tibble, we can easily pass it to dplyr functions to create custom summaries. For example, we can look at the minimum, median, and maximum ROC AUC values that occurred within each cross validation fold during tuning.
To do this, we filter for dot-metric being roc_auc, then group by the id column which has the fold identifier values, and calculate summary statistics of the dot-estimate column with the summarize() function. Here we are looking to see if the estimated values are fairly consistent across folds. Wild fluctuations would be an indicator of model overfitting.
4. Viewing the best performing models
To make exploring tuning results easier, we can use the show_best() function to display the top performing hyperparameter combinations.
This function takes our dt_tuning object as the first argument, the metric for evaluation as the second, and the number of model combinations to display.
From the results, the hyperparameter combination labeled as Model1 in the dot-config column has the largest average ROC AUC.
5. Selecting a model
The select_best() function is used to select the best performing hyperparameters.
We pass our dt_tuning object to select_best() and provide the metric on which to evaluate performance.
This function returns a tibble with the hyperparameter values that produced the largest average performance metric value. For our results, it was Model1 for the ROC AUC metric.
6. Finalizing the workflow
The results of select_best() can be used to finalize a workflow object with the finalize_workflow() function.
To finalize a workflow, pass it to the finalize_workflow() function and provide the results of the select_best() function as an argument. In our case, we pass the leads_tune_wkfl to finalize_workflow() and provide the best_dt_model tibble.
This returns an updated workflow object with the model hyperparameter values set to the optimal values found during the tuning process.
7. Model fitting
After a workflow has been finalized, it can be trained with the last_fit() function and the original data split object, leads_split in our example.
Behind the scenes, the training and test datasets are created from leads_split, our recipe is trained and applied, our tuned decision tree model is trained with the entire training dataset, and predictions and metrics are generated with the test dataset. For our model, we got a ROC AUC of 0-point-793 on the test dataset, which was held out of the process until this final step. This is in line with our tuning results, which had ROC AUC values near 0-point-8 for most combinations. In general, having similar performance between cross validation and the test dataset indicates that a model will perform similarly on new data sources.
8. Let's practice!
Let's practice finalizing workflows based on tuning results!