Introduction to model validation
1. Introduction to model validation
Hello, my name is Kasey Jones - And welcome to this course on model validation. Let's get started!2. What is model validation?
So what is model validation? Well, model validation consists of various steps and processes that ensure your model performs as expected on new data. The most common way to do this is to test your model's accuracy on data it has never seen before (called a holdout set). If your model's accuracy is similar for the data, it was trained on, and the holdout data, you can claim that your model is validated. However, model validation can also consist of choosing the right model, the best parameters, and even the best accuracy metric. The ultimate goal of model validation is to end up with the best performing model possible, that achieves high accuracy on new data. Before we begin exploring model validation, let's review some basic modeling steps using scikit-learn.3. scikit-learn modeling review
Modeling in Python follows a simple procedure, regardless of the type of model you are constructing. Whether you are a seasoned scikit-learn veteran or new to building models with this module, let's take a quick look at these steps. First, we create a model by specifying the model type and its parameters. In this case, we are creating a random forest regression model with RandomForestRegressor(). Second, we fit the model using the dot-fit() method. This method has two main arguments. X, an array of data used in the model as training data, and y, an array of response values matching the size of the X array. When dot-fit() is used, the model parameters will be printed in the console.4. Modeling review continued
To assess model accuracy, we generate predictions for data using the dot-predict() method. And lastly, we look at the accuracy metrics. Here we are comparing the model's predictions (the variable predictions) and the actual responses, y_test. Future lessons and exercises will be devoted to accuracy metrics, as they are a vital component to model validation. For this current example though, we are looking at the mean absolute error. This function takes two arrays as arguments. The true values, y_true, and the predicted values, y_pred, and returns the mean absolute error between them.5. Review prerequisites
This process of generating a model, fitting, predicting, and then reviewing model accuracy will be repeated throughout this course. If you are unfamiliar with these steps, you should consider taking the prerequisite courses. They will go into more detail about using Python and performing these modeling steps.6. fivethirtyeight's candy dataset
Throughout this course, we will use fivethirtyeight's ultimate Halloween candy power ranking dataset several times. This dataset contains 85 different candies, data on their various characteristics, and a column specifying how often that candy was selected in a head-to-head match-up with other candies. This column is a win-percentage and contains values between 0 and 100.7. Seen vs. unseen data
Model validation's main goal is to ensure that a predictive model will perform as expected on new data. Obtaining predictions for training data (or seen data) and testing data (or unseen data) is coded in the same way and uses the predict() method. Generally, models perform a lot better on data they have seen before, as unseen data may have features or characteristics that were not exposed in the model. If your training and testing errors are vastly different, it may be a sign that your model is overfitted. We will use model validation to make sure we get the best testing error possible.8. Let's begin!
Let's see why model validation is so important by looking at an example of training and testing accuracies.Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.