Grid and random search with mlr

1. Grid and random search with mlr

Now that you have the basics of machine learning with the mlr package fresh on your mind, let's look into hyperparameter tuning with grid and random search in mlr.

2. Hyperparameter tuning with mlr

For hyperparameter tuning in mlr you have to define three things: - the search space, this means the hyperparameter values you want to compare and tune - the tuning method, which could be grid or random search - the resampling scheme Let's look at each step in turn.

3. Defining the search space

First, we will define the hyperparameter search space for our learner. To do this, we use the makeParamSet function. Within this main function, we have several support functions for defining different types of parameter spaces: - makeNumericParam let's us define numeric ranges between specified values - makeIntegerParam does the same just with integers - makeDiscreteParam is for defining discrete lists of values to search - makeLogicalParam let's us define TRUE/FALSE values - and makeDiscreteVectorParam for vectors of discrete parameters Now, let's look at what hyperparameters we have in a deep learning model. If we enter the learner class into the getParamSet function, it will return a table with hyperparameters for this function and tell us what type they are, what default values will be set, their lower and upper boundaries, if they are tunable, plus some additional information.

4. Defining the search space

So, let's take the makeParamSet function and create entries for the hyperparameters hidden, activation and l1 and l2.

5. Defining the tuning method

Next, we define the tuning method. The two basic versions you can choose from are grid search, where every combination of hyperparameters will be compared, and random search, where a subset of our specified values will be tested randomly. The two respective functions are makeTuneControlGrid and makeTuneControlRandom. When we look at the objects we created with these two functions, it will print a summary to the console that tells us what settings were chosen (in our case, the default values). The maxit argument of the makeTuneControlRandom function determines the number of iterations for random search - per default 100. One thing is important to note: Grid search can only deal with discrete parameter sets, defined with the makeDiscreteParam function, while random search can deal with all types of parameter sets.

6. Define resampling strategy

The final step is to define the resampling strategy. With the makeResampleDesc function you can create a description object for a resampling strategy, like cross-validation, repeated cross-validation, leave-one-out, bootstraping or holdout. Alternatively, you can store a set of integer vectors for training and test sets with the makeResampleInstance function. Here, we will use 3 x 5-fold repeated cross-validation to measure the performance of a specific parameter combination. If we want to predict training AND validation data (in mlr called test) during resampling to detect overfitting, we choose predict = both. We now have these three objects in R: - cross_val - param_set - and ctrl_grid So, now we take our task and learner object and combine all in the tuneParams function, which will perform the hyperparameter tuning and model fitting.

7. Tuning hyperparameters

We could additionally define performance evaluation metrics but if we don't, mlr will use the Mean misclassification error rate (mmce) by default for classification tasks. See getDefaultMeasure to find the default measure for a task. Here, you see the output that is printed to the console during training. We will get feedback for every hyperparameter that was tested, as well as the time it took to run. The last row returns the best hyperparameter result. In this case 2 hidden layers, Tanh activation and l1=0.11 and l2=0.1. The entire training process took about 30 seconds.

8. Let's practice!

Now, it's your turn again!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.