Batch training

1. Batch training

Earlier in the chapter, we learned how to train a linear model to predict house prices. In this video, we will use batch training to handle large datasets.

2. What is batch training?

So what is batch training exactly? To answer this, let's return to the linear model you used to predict house prices earlier in the chapter. But this time, let's say the dataset is much larger and you want to perform the training on a GPU, which has only small amount of memory. Since you can't fit the entire dataset in memory, you will instead divide it into batches and train on those batches sequentially. A single pass over all of the batches is called an epoch and the process itself is called batch training. It will be quite useful when you work with large image datasets. Beyond alleviating memory constraints, batch training will also allow you to update model weights and optimizer parameters after each batch, rather than at the end of the epoch.

3. The chunksize parameter

Earlier, we discussed using pandas to load data with read csv. The same function can be used to load data in batches. If, for instance, we have a 100 gigabyte dataset, we might want to avoid loading it all at once. We can do this by using the chunksize parameter. The code block shows how this can be done. Let's first import pandas and numpy. Instead of loading the data in a single one-liner, we'll write a for loop that iterates through the data in steps of 100 examples. Each 100 will be available as batch, which we can use to extract columns, such as price and size in the housing dataset. We can then convert these to numpy arrays and use them to train.

4. Training a linear model in batches

We now know how to load data from csv files in fixed-size batches using pandas. This means that we can handle data sets of tens or even hundreds of gigabytes without exceeding the memory constraints of our system. Let's look at a minimal example with a linear model using the King County housing dataset. We will start by loading tensorflow, pandas, and numpy. Next, we'll define variables for the intercept and slope, along with the linear regression model.

5. Training a linear model in batches

We then define a loss function, which takes the slope and intercept, and two sources of data: the features and the targets. It then returns the mean squared error loss. After defining the loss function, we instantiate an adam optimizer, which we will use to perform minimization.

6. Training a linear model in batches

The next step is to train the model in batches. Again, we do this by using a for loop and supplying a chunksize to the read csv function. Note that we take each batch, separate it into features and a target, convert those into numpy arrays, and then pass them to the minimize operation. Within the minimize operation, we pass the loss function as a lambda function and we supply a variable list that contains only the trainable parameters, intercept and slope. This loop will continue until we have stepped through all of the examples in csv read. Importantly, we did not ever need to have more than 100 examples in memory during the entire process. Finally, we print our trained intercept and slope. Note that we did not use default argument values for input data. This is because our input data was generated in batches during the training process.

7. Full sample versus batch training

So what is the value of batch training? When we trained with the full sample, we updated the optimizer and model parameters once per training epoch and passed data to the loss function without modification, but were limited by memory constraints. With batch training, we updated the model weights and optimizer parameters multiple times each epoch and divided the data into batches, but no longer faced any memory constraints. In later chapters, you'll automate batch training by using high level APIs. Importantly, however, high level APIs will not typically load the sample in batches by default, as we have done here.

8. Let's practice!

That was a lot of information! Let's break this down into simple steps with some exercises.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.