Overview of XGBoost's hyperparameters

1. Tunable parameters in XGBoost

Let's now go over the differences in what parameters can be tuned for each kind of base model in XGBoost. The parameters that can be tuned are significantly different for each base learner.

2. Common tree tunable parameters

For the tree base learner, which is the one you should use in almost every single case, the most frequently tuned parameters are outlined below. The learning rate affects how quickly the model fits the residual error using additional base learners. A low learning rate will require more boosting rounds to achieve the same reduction in residual error as an XGBoost model with a high learning rate. Gamma, alpha, and lambda were described in chapter 2 and all have an effect on how strongly regularized the trained model will be. Max_depth must a positive integer value and affects how deeply each tree is allowed to grow during any given boosting round. Subsample must be a value between 0 and 1 and is the fraction of the total training set that can be used for any given boosting round. If the value is low, then the fraction of your training data used would per boosting round would be low and you may run into underfitting problems, a value that is very high can lead to overfitting as well. Colsample_bytree is the fraction of features you can select from during any given boosting round and must also be a value between 0 and 1. A large value means that almost all features can be used to build a tree during a given boosting round, whereas a small value means that the fraction of features that can be selected from is very small. In general, smaller colsample_bytree values can be thought of as providing additional regularization to the model, whereas using all columns may in certain cases overfit a trained model.

3. Linear tunable parameters

For the linear base learner, the number of tunable parameters is significantly smaller. You only have access to l1 and l2 regularization on the weights associated with any given feature, and then another regularization term that can be applied to the model's bias. Finally, its important to mention that the number of boosting rounds (that is, either the number of trees you build or the number of linear base learners you construct) is itself a tunable parameter.

4. Let's get to some tuning!

Now that we've covered the parameters that are usually tuned when using XGBoost, lets get to some tuning!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.