1. Measuring the Test Performance
Throughout this chapter you've worked with a classic machine learning workflow.
2. Machine Learning Workflow
The first step of this workflow was to split your data into two sections, train and test.
3. Machine Learning Workflow
The test portion was intentionally held out in order to evaluate the final model with an independent set of data.
4. Machine Learning Workflow
The train portion of the data was further split into iterative sections of train and validate using cross validation for the purpose of model selection.
5. Machine Learning Workflow
Each train portion was used to build a model and the held out validate portion was used to evaluate it.
6. Machine Learning Workflow
Resulting in measures of validation performance for each cross validation fold for each model and hyperparameter.
Aggregating the validation performance for each model allowed us to compare multiple models as well as their respective hyper parameters
7. Machine Learning Workflow
in order to select the model-hyperparameter combination with the best overall performance.
For the gapminder dataset, the best performing model was the random forest model with a hyperparameter mtry of 4.
8. Machine Learning Workflow
This brings us to the final section of this workflow, building and evaluating our final model.
Now, you will use all of the train data prepared during the initial split to build the random forest model. This is the final model and is the one you would expect to use in a production environment.
9. Machine Learning Workflow
As such you would like to know how well you can expect this model will perform on new data. To do this you bring back the test data that was intentionally ignored thus far and treat it as the desired new data for evaluation.
By comparing the actual values of life expectancy for the test set with the values predicted using the final model you can estimate the model's performance on new data. This is known as the model's test performance.
10. Measuring the Test Performance
To perform these steps in R you first build the best performing model, which in this case was the random forest model built using ranger with an mtry value of 2 and 100 trees.
Next, you prepare the actual and predicted values for comparison.
Finally, you need to compare the actual and predicted values using a desired metric, in this case the mean absolute error.
11. Let's practice!
Let's see how well the final model performed.