Why distributed training is needed
1. Why distributed training is needed
Person: In this module, we'll explore how to run a distributed training job with TensorFlow. We'll begin with understanding why distributed training is needed. Then we'll explore distributed training architectures. And lastly, we'll provide an overview of TensorFlow distributed training strategies. Deep learning works because datasets are large. Notice that the X-axis here is logarithmic. For every doubling in the size of the data, the error rate falls linearly. A more complex model also helps-- that is, the jump from the blue line to the orange line. But more data is even more helpful in this situation. As a consequence of both of these trends, in terms of larger data sizes and more complex models, the compute required to build state-of-the-art models has grown over time. This growth is exponential as well. Each Y-axis tick on this graph shows a 10x increase in computational need. AlexNet--which started the deep learning revolution in 2013-- required less than 0.01 petaflops per second-day in compute per day for training. By the time you get to Neural Architecture Search-- the learn-to-learn model published by Google in 2017-- you need about 100 petaflops per second-day, or 1,000 times more compute than you needed for AlexNet. The growth in algorithm complexity and data size means that with complex models and large data volumes distributed systems are pretty much a necessity when it comes to machine learning. Training complex networks with large amounts of data can often take a long time. This graph shows training time on the X-axis plotted against the accuracy of predictions on the Y-axis when training an image recognition model on a GPU. As the dotted line shows, it took around 80 hours to reach 75% accuracy. If your training takes a few minutes to a few hours, it will make you productive and happy, and you can try out different ideas fast. If the training takes a few days, you can still deal with that by running a few ideas in parallel. If the training starts to take a week or more, your progress will slow down because you can't try out new ideas quickly. And if it takes more than a month, well, that's probably not even worth thinking about. And this is no exaggeration. Training deep neural networks such as ResNet-50 can take up to a week on one GPU. A natural question to ask is how can you make training faster? You can use a more powerful device such as a TPU or GPU accelerator. You could optimize your input pipeline. Or you can try out distributed training. In the next video, we'll explore distributed training architectures.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.