Comparing Grid and Random Search

1. Comparing Grid and Random Search

In the previous lessons we looked at both Grid and Random search in depth. Before we move on to more advanced methods, it is a good idea to stop and reinforce our knowledge on these two valuable techniques.

2. What's the same?

There are a number of similarities for these techniques. Both are automated ways of tuning different hyperparameters. For both you set the grid to sample from (which hyperparameters and values for each). For both you need to think carefully about which hyperparameters and values to sample from as your model will only be as good as the grid you set! And for both you set a cross-validation scheme and scoring function.

3. What's different?

There are also some key differences between these two techniques. Grid search exhaustively tries all the combinations in the grid or sample space and therefore has no sampling methodology. Random search tries a subset of combinations randomly and you can additionally define how to sample. Remember the default in Scikit Learn is to have every combination equally likely to be chosen, also known as uniform distribution. Because of this, grid search is very computationally expensive compared to random search But Grid Search is guaranteed to find the best score in the sample space whereas random search is not. Though random search has a higher chance to find a good score faster.

4. Which should I use?

So which technique should you use? As we often say in data science - 'It depends'. However there are some things you can consider. Firstly, the more data you have, the stronger the argument for random search. More hyperparameters and values to try also means random search might be a better option. Additionally if you don't have a lot of time or computing power, random search will be more economical. Remember, random search has more chance of getting a good result faster, even if not the absolute best.

5. Let's practice!

Let's do some exercises to illustrate the differences between grid and random search visually and conceptually.

This exercise is part of the course

Hyperparameter Tuning in Python

IntermediateSkill Level

4.9+

Start Course for Free

In this introductory chapter you will learn the difference between hyperparameters and parameters. You will practice extracting and analyzing parameters, setting hyperparameter values for several popular machine learning algorithms. Along the way you will learn some best practice tips & tricks for choosing which hyperparameters to tune and what values to set & build learning curves to analyze your hyperparameter choices.

Exercise 1: Introduction & 'Parameters'Exercise 2: Parameters in Logistic Regression Exercise 3: Extracting a Logistic Regression parameter Exercise 4: Extracting a Random Forest parameter Exercise 5: Introducing Hyperparameters Exercise 6: Hyperparameters in Random Forests Exercise 7: Exploring Random Forest Hyperparameters Exercise 8: Hyperparameters of KNN Exercise 9: Setting & Analyzing Hyperparameter Values Exercise 10: Automating Hyperparameter Choice Exercise 11: Building Learning Curves

This chapter introduces you to a popular automated hyperparameter tuning methodology called Grid Search. You will learn what it is, how it works and practice undertaking a Grid Search using Scikit Learn. You will then learn how to analyze the output of a Grid Search & gain practical experience doing this.

Exercise 1: Introducing Grid Search Exercise 2: Build Grid Search functions Exercise 3: Iteratively tune multiple hyperparameters Exercise 4: How Many Models?Exercise 5: Grid Search with Scikit Learn Exercise 6: GridSearchCV inputs Exercise 7: GridSearchCV with Scikit Learn Exercise 8: Understanding a grid search output Exercise 9: Using the best outputs Exercise 10: Exploring the grid search results Exercise 11: Analyzing the best results Exercise 12: Using the best results

In this chapter you will be introduced to another popular automated hyperparameter tuning methodology called Random Search. You will learn what it is, how it works and importantly how it differs from grid search. You will learn some advantages and disadvantages of this method and when to choose this method compared to Grid Search. You will practice undertaking a Random Search with Scikit Learn as well as visualizing & interpreting the output.

Exercise 1: Introducing Random Search Exercise 2: Randomly Sample Hyperparameters Exercise 3: Randomly Search with Random Forest Exercise 4: Visualizing a Random Search Exercise 5: Random Search in Scikit Learn Exercise 6: RandomSearchCV inputs Exercise 7: The RandomizedSearchCV Object Exercise 8: RandomSearchCV in Scikit Learn Exercise 9: Comparing Grid and Random Search

Current Exercise

Exercise 10: Comparing Random & Grid Search Exercise 11: Grid and Random Search Side by Side

In this final chapter you will be given a taste of more advanced hyperparameter tuning methodologies known as ''informed search''. This includes a methodology known as Coarse To Fine as well as Bayesian & Genetic hyperparameter tuning algorithms. You will learn how informed search differs from uninformed search and gain practical skills with each of the mentioned methodologies, comparing and contrasting them as you go.

Exercise 1: Informed Search: Coarse to Fine Exercise 2: Visualizing Coarse to Fine Exercise 3: Coarse to Fine Iterations Exercise 4: Informed Search: Bayesian Statistics Exercise 5: Bayes Rule in Python Exercise 6: Bayesian Hyperparameter tuning with Hyperopt Exercise 7: Informed Search: Genetic Algorithms Exercise 8: Genetic Hyperparameter Tuning with TPOT Exercise 9: Analysing TPOT's stability Exercise 10: Congratulations!