1. Evaluating hyperparameters with mlr
In this lesson, I will show different ways to plot and evaluate models with different hyperparameters.
2. Evaluation of our results can tell us:
Generally, we want to evaluate hyperparameter tuning to assess:
- How different hyperparameters affect model performance.
- Which hyperparameters have particularly strong or weak impact.
- Whether our hyperparameter search converged, i.e. whether we can be reasonably confident that we found the most optimal combination (or close to it).
3. Recap
Let's look at our former example: hyperparameter tuning with deep learning from the h2o package.
Let's set up the tuning just as before, before we discuss how to evaluate the results: create a grid of hyperparameters,
define tune control,
resampling strategy,
task and the learner.
A small variation is our resampling scheme: this time, I am using a holdout set, which is much faster to train than repeated cross validation. The default split is 2 / 3.
4. Evaluating the tuning results
Here, you see the tuning result, i.e. the best combination of hyperparameters from our random set: 1 hidden layer with 10 nodes, Rectifier activation and l1 and l2 regularization of 0.541 and 0.229.
Mean misclassification error was 0.16, which is okay.
With the generateHyperParsEffectData function, we generate the results object.
Because we are tuning more than 2 hyperparameters, we need to set partial dependence to TRUE.
The output of this function gives a summary of the settings: which hyperparameters were tuned, which measure was used to evaluate them, what optimizer was used and whether we used nested cross validation.
And we get a table with different hyperparameters that were tested. In our case, we see five randomly picked hyperparameter combinations, what values were chosen for each, as well as the mmce and the execution time. The entire table can be called with generateHyperParsEffectData$data.
5. Plotting hyperparameter tuning results
The hyperparameter effect data can also be plotted with the plotHyperParsEffect function.
A plot can often make it easier to grasp the overall information and evaluate how well different hyperparameters performed in our model.
The function takes a few inputs:
- the generated hyperparameter effect data
- a regression method to calculate partial dependence. Here, I am choosing random Forest but any regression method from the mlr repertoire can be used.
- The x axis can show any of our hyperparameters. Here I chose l1. The y axis can show the remaining metrics, like mmce or iteration. Optionally, you can choose a z variable, in this case the number of hidden layers which is shown with different colors.
- By default, plotHyperParsEffect will create a scatter plot, we can change that to a line plot, heatmap or contour plot, with the plot.type argument.
6. Now it's your turn!
Alright, now it's your turn to evaluate hyperparameters!