Hyperparameter optimization with Optuna
1. Hyperparameter optimization with Optuna
Throughout this course, we have introduced a large number of hyperparameters.2. What are hyperparameters
These are variables that affect the performance of the model, but cannot be learned by the model. Here are some examples. The discount rate applied to the rewards obtained during an episode. In PPO, the clipping epsilon and entropy bonus. In DQN, the experience replay buffer size and the batch size; the epsilon greediness decay schedule; the tau parameter of fixed Q targets. The optimizer learning rate. The network architecture: number of hidden layers, and number of nodes in each layer. As the number of hyperparameters increases, choosing them becomes a challenge.3. How to choose hyperparameter values
Hyperparameter search is the process of finding the hyperparameters values that maximize the objective we care about. In DRL, this may be the average cumulative rewards of the trained agent over 100 episodes. Search techniques include: informally picking values by hand; systematically exploring all regions in the range of possible hyperparameters, or grid search; sampling random values for the hyperparameters, or random search; and dedicated algorithms that aim to make smarter decisions about which hyperparameter values to try next.4. Optuna
A number of tools and packages can help in this process. We use Optuna, an approachable and powerful open source package. Optuna supports different samplers, with TPE being the default. We first import the optuna package. We define an objective function. We will get back to this next. We then instantiate a study, the data structure used to store the data about the tuning process. The study.optimize method kicks off the hyperparameter tuning process, one trial at a time. Afterwards, we can read the optimal hyperparameter values with study.best_params.5. Specifying the objective function
The objective function is at the core of the Optuna optimization process. Let's go over an example. We first specify which hyperparameters the trials should optimize over. For a real-valued hyperparameter, this can be done with trial.suggest_float; the arguments are the name of the hyperparameter, and the lower and upper bounds for the search: in this case, hyperparameters x and y range between -10 and 10. The objective function returns the metric that we are trying to minimize: here, a parabola with its minimum at x=2, y=-3. This offers full flexibility. Any hyperparameter can be defined: floats, integers, or categorical.6. The optuna study
By default, optuna saves the study in memory. In practice, it is often preferable to store it; using an sqlite database is the simplest way. Specify the database as the storage argument of `create_study`. When we run study.optimize with n_trials=100, Optuna samples 100 values for x and y between -10 and 10. Initially, the sampling is at random, uniformly between -10 and 10. Over time, as trial results come in, Optuna explores the hyperparameter space efficiently, spending more time in promising areas. If you do not specify n_trials, Optuna will keep running trials until the Python session is interrupted or terminated. You can load the study later on in a separate session using optuna.load_study.7. Exploring the study results
Though we won't cover it here in detail, Optuna includes a visualizations module, giving insights into the influence of each hyperparameter. Among others, this includes a hyperparameter importance plot and a contour plot. Analyzing these and other plots can help identify a good set of hyperparameters to use.8. Let's practice!
Let's practice!Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.