Mirrored strategy

1. Mirrored strategy

person: Mirrored strategy is the simplest way to get started with distributed training. You can use mirrored strategy when you have a single machine with multiple GPU devices. Mirrored strategy will create a replica of the model on each GPU. During training, one minibatch is split into n parts, where "n" equals the number of GPUs, and each part is fed to one GPU device. For this setup, mirrored strategy manages the coordination of data distribution and gradient updates across all of the GPUs. Let's look at an image classification example where a Keras ResNet model with the functional API is defined. First, download the Cassava dataset from TensorFlow datasets. Then add a preprocess_data function to scale the images. From there, map, shuffle, and prefetch the data, and then define the model. Let's create the strategy object using tf.distribute. MirroredStrategy. Next, let's create the model with variables within the strategy scope. These variables include the model spare_categorical_crossentropy for loss, the Keras optimizer, and metrics variables to compute accuracy. The last change you'll want to make is to the batch size. When you carry out distributed training with the tf.distribute strategy API and tf.data, the batch size now refers to the global batch size. In other words, if you pass a batch size of 64 and you have two GPUs, then each machine will process 32 examples per step. In this case, 64 is known as the global batch size, and 32 is the per replica batch size. To make the most out of your GPUs, you'll want to scale the batch size by the number of replicas. From there, map, shuffle, and prefetch the data. You then call model.fit on the training data. Here we're going to run five passes of the entire training dataset. Now let's explain what actually happens when we call model.fit before adding a strategy. For simplicity, imagine you have a simple linear model instead of the ResNet 50 architecture. In TensorFlow, you can think of this simple model in terms of its computational graph or directed acyclic graph, here referred to as a DAG. Here the matmul op takes in the x and W tensors, which are the training batch and weights, respectively. The resulting tensor is then passed to the add op with the tensor b, which is the model's bias terms. The result of this op is ypred, which is the model's predictions. Now this is an example of data parallelism with two GPUs. The input batch, x, is split in half, and one slice is sent to GPU 0 and the other to GPU 1. In this case, each GPU calculates the same ops, but on different slices of the data. For more information on optimization, please refer to the guide titled, "Optimize TensorFlow GPU performance using the Profiler" at tensorflow.org/guide/profiler.

2. Let's practice!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.