Evaluate the grid

Earlier in the chapter we split the dataset into three parts: training, validation and test.

A dataset that is not used in training is sometimes referred to as a "holdout" set. A holdout set is used to estimate model performance and although both validation and test sets are considered to be holdout data, there is a key difference:

Just like a test set, a validation set is used to evaluate the performance of a model. The difference is that a validation set is specifically used to compare the performance of a group of models with the goal of choosing a "best model" from the group. All the models in a group are evaluated on the same validation set and the model with the best performance is considered to be the winner.
Once you have the best model, a final estimate of performance is computed on the test set.
A test set should only ever be used to estimate model performance and should not be used in model selection. Typically if you use a test set more than once, you are probably doing something wrong.

Write a loop that evaluates each model in the grade_models list and stores the validation RMSE in a vector called rmse_values.
The which.min() function can be applied to the rmse_values vector to identify the index containing the smallest RMSE value.
The model with the smallest validation set RMSE will be designated as the "best model".
Inspect the model parameters of the best model.
Generate predictions on the test set using the best model to compute test set RMSE.

Exercise

Evaluate the grid

Instructions