Informed Search: Coarse to Fine
1. Informed Search: Coarse to Fine
In this final chapter, we will look at a group of hyperparameter tuning techniques known as informed search. We will begin with a basic informed search methodology, Coarse to Fine.2. Informed vs Uninformed Search
So far, everything we have done with grid and random search were instances of uninformed search. When performing uninformed search, each iteration of hyperparameter tuning (each model) does not learn from the previous iterations. This allows us to undertake hyperparameter tuning in parallel. However, doing so much work before being able to learn and iterate seems a bit inefficient, doesn't it?3. Informed vs Uninformed
Let's make it more explicit. So far, we have been creating all the models at once and collating their scores before deciding the best at the end. An alternate approach would be to build models sequentially, learning from each iteration. This is what undertaking hyperparameter searching in an informed way allows us to do.4. Coarse to Fine Tuning
A first, basic, informed search methodology is known as 'Coarse to Fine tuning'. The name comes from the fact that you start out with a rough approach and iteratively refine your hyperparameter search. The process goes as follows. First, undertake a random search. Second, review results to see promising areas in your hyperparameter search space. Then, undertake a grid search in the smaller area. Continue this process until an optimal score is obtained or the area becomes too small (meaning not many hyperparameter values) to properly search. You could also substitute step 3 with further random searches before the grid search.5. Why Coarse to Fine?
Coarse to Fine tuning optimizes the advantages of grid and random search. It uses the advantages of both methods. The wide searching abilities of random search and including a grid search for deeper coverage at the relevant time. It therefore better utilizes efforts so you can iterate. If you see one area on the grid is producing poor results, you will undertake your next search elsewhere, saving you effort. Also note that Coarse to fine is not informed on every model, but on batches of models. Still, it is more informed than what we have done so far!6. Undertaking Coarse to Fine
Let us set up a pretty large hyperparameter search over these following ranges. How many possible hyperparameter combinations are there? If we create the combinations as we did in previous exercises we can see 134,400 possible combinations!7. Visualizing Coarse to Fine
Rather than running all those models, we can do a random search on just 500 models Let's first visualize just the accuracy column as a density plot. Wow! There were certainly some bad models in there. Good thing we didn't do a huge grid search and waste our time on those! But there are some good models in there, what hyperparameters did they use?8. Visualizing Coarse to Fine
Here is a snapshot of the top models of the results DataFrame. Hmmmm, that's not very informative about what to do next, is it? Perhaps the min_samples_leaf being 7? But we don't know if there were also bad models with this same value.9. Visualizing Coarse to Fine
Instead, let's visualize each hyperparameter against the accuracy scores to see if we can see any trends. We will do a simple scatter plot using a results data frame. That's more interesting! See how accuracy tends to be lower when max_depth is below 6 or 7 and above 30?10. Visualizing coarse to Fine
A similar exercise with the other two hyperparameters finds some interesting insights. The min_samples_leaf being a bit better below 8, and the learn_rate seems worse above 1-point-3. Now we have a plan for our next iteration.11. The next steps
Let's summarize what we learned from the first grid search. Values of max_depth between 8 and 30, learn_rate of less than 1-point-3 and perhaps min_samples_leaf less than 8 tend to have higher accuracies. You can take this and undertake a more refined grid search or even another random search around these values to improve the model This simple, bivariate graphing was insightful, but using more advanced visualization techniques could also prove useful.12. Let's practice!
Let's practice undertaking a Coarse to Fine search!Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.