Model validation

1. Model validation

You may recall from previous machine learning classes that your model's performance on the training data is not a good indication of how it will perform on new data. For this reason, we use validation data to test model performance. Validation data is data that is explicitly held out from training, and used only to test model performance.

2. Validation in deep learning

You may already be familiar with k-fold cross validation. In practice, few people run k-fold cross validation on deep learning models because deep learning is typically used on large datasets. So the computational expense of running k-fold validation would be large, and we usually trust a score from a single validation run because those validation runs are reasonably large. Keras makes it easy to use some of your data as validation data, and we see that

3. Model validation

in this code, where we specify the split using the keyword argument validation_split when calling the fit method. Here, we have already specified a model, and we'll make small changes in both the compile and fit steps to see model validation information. This is a classification problem, and we'd like to see measures of accuracy. So, we include metrics equals 'accuracy' in the compile step. In the fit step, we specify what fraction of the data is used for validation. In this case, we'll use 30%.Our goal is to have the best validation score possible, so we should keep training while validation score is improving, and then stop training when the validation score isn't improving. We do this with something called

4. Early Stopping

"early stopping." We can use early stopping with only some small changes to the code. See here, we've imported something called EarlyStopping. We then create an "early stopping monitor" before fitting the model. That monitor takes an argument called patience, which is how many epochs the model can go without improving before we stop training. 2 or 3 are reasonable values for patience. Sometimes you'll get a single epoch with no improvement, but the model will start improving again after that epoch. But if you see 3 epochs with no improvement, it's unlikely to turn around and start improving again. We pass early_stopping_monitor to the fit function under an argument called callbacks. Notice that callbacks takes a list. You may consider adding other callbacks as you become very advanced. But early stopping is all you want for now. By default, keras trains for 10 epochs. Now that we have smart logic for determining when to stop, we can set a high maximum number of epochs. This happens with the epochs argument, as you see here. Keras will go until this number of epochs, unless the validation loss stops improving, in which case it will stop earlier. This is smarter training logic than relying on a fixed number of epochs without looking at the validation scores.

5. Output from early stopping

Let's look at the output. In epoch 9 we had a validation loss score of point-6513. We didn't beat that score in the next 2 epochs, so we stopped training. Now that you have a reliable way of measuring model performance, namely through scores, you should feel free

6. Experimentation

to experiment with different architectures. More layers, fewer layers. Layers with more nodes, layers with fewer nodes. And so on. Creating a great model requires some experimentation. Before we finish, we'll give a little bit of insight into how to choose where you experiment. -

7. Let's practice!

But, now that you can get validation scores, you are poised to run those experiments and figure out what works best for your data.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.