You'll prepare data for distributed training by splitting the data across multiple devices and copying the model on each device. Accelerator provides a convenient interface for data preparation, and you'll learn how to preprocess images, audio, and text as a first step in distributed training.

Prepare models with AutoModel and Accelerator

Loading and inspecting pre-trained models

Automatic device placement with Accelerator

Preprocess images and audio for training

Preprocess image datasets

Preprocess audio datasets

Prepare datasets for distributed training

Preprocess text for training

Preprocess text with AutoTokenizer

Save and load the state of preprocessed text

Data Preparation with Accelerator

In distributed training, each device trains on its data in parallel. You'll investigate two methods for distributed training: Accelerator enables custom training loops, and Trainer simplifies the interface for training.

Fine-tune models with Trainer

Define evaluation metrics

Specify the TrainingArguments

Set up the Trainer

Train models with Accelerator

Prepare a model for distributed training

Training loops before and after Accelerator

Building a training loop with Accelerator

Evaluate models with Accelerator

Setting the model in evaluation mode

Logging evaluation metrics

Distributed Training with Accelerator and Trainer

Distributed training strains resources with large models and datasets, but you can address these challenges by improving memory usage, device communication, and computational efficiency. You'll discover the techniques of gradient accumulation, gradient checkpointing, local stochastic gradient descent, and mixed precision training.

Gradient accumulation

Gradient accumulation with Accelerator

Gradient accumulation with Trainer

Gradient checkpointing and local SGD

Gradient checkpointing with Accelerator

Gradient checkpointing with Trainer

Local SGD with Accelerator

Mixed precision training

Mixed precision training with basic PyTorch

Mixed precision training with Accelerator

Mixed precision training with Trainer

Improving Training Efficiency

You'll focus on optimizers as levers to improve distributed training efficiency, highlighting tradeoffs between AdamW, Adafactor, and 8-bit Adam. Reducing the number of parameters or using low precision helps to decrease a model's memory footprint.

Balanced training with AdamW

AdamW with Trainer

AdamW with Accelerator

Compute the optimizer size

Memory-efficient training with Adafactor

Adafactor with Trainer

Adafactor with Accelerator

Mixed precision training with 8-bit Adam

Set up the  8-bit Adam optimizer

8-bit Adam with Trainer

8-bit Adam with Accelerator

Options

AdamW

Adafactor

8-bit Adam

Which optimizer is it?

Congratulations!

Training with Efficient Optimizers

Audio dataset

Crop image

Agricultural QA dataset

MRPC dataset

Do you want your machine learning models to train in a fraction of the time? Distributed training allows you to train models across multiple devices, making it possible to scale your AI projects and reduce training times. In this course, you'll learn how to use the Hugging Face Accelerator library and PyTorch to prepare models for distributed training on your device. You'll explore data sharding, model replication, and parameter synchronization while gaining hands-on experience applying these techniques to real-world workflows, whether working on image, text, or audio models.

Distributed training is an essential skill in large-scale machine learning, helping you to reduce the time required to train large language models with trillions of parameters. In this course, you will explore the tools, techniques, and strategies essential for efficient distributed training using PyTorch, Accelerator, and Trainer. 
<h2>Preparing Data for Distributed Training</h2>
You'll begin by preparing data for distributed training by splitting datasets across multiple devices and deploying model copies to each device. You'll gain hands-on experience in preprocessing data for distributed environments, including images, audio, and text.
<h2>Exploring Efficiency Techniques</h2>
Once your data is ready, you'll explore ways to improve efficiency in training and optimizer use across multiple interfaces. You'll see how to address these challenges by improving memory usage, device communication, and computational efficiency with techniques like gradient accumulation, gradient checkpointing, local stochastic gradient descent, and mixed precision training. You'll understand the tradeoffs between different optimizers to help you decrease your model's memory footprint.

By the end of this course, you'll be equipped with the knowledge and tools to build distributed AI-powered services.

Intermediate Deep Learning with PyTorch

Working with Hugging Face

Learn how to reduce training times for large language models with Accelerator and Trainer for distributed training

Efficient AI Model Training with PyTorch | Distributed training for LLMs 

Efficient AI Model Training with PyTorch

Computes the exponential moving average of the gradients and squared gradients in high precision

Common default optimizer for training models

Stores the exponential moving average of the squared gradients only

Reduces the number of parameters to store during optimization but still uses high precision

Computes the exponential moving average of the gradients and squared gradients in low precision

Stores two bytes per parameter, about one fourth as large as AdamW

Which optimizer is it?

Efficient AI Model Training with PyTorch

Hands-on interactive exercise