1. Learn
    2. /
  2. Courses
    3. /
  3. Distributed AI Model Training in Python
  • 1

    Data Preparation with Accelerator

    Free

    You'll prepare data for distributed training by splitting the data across multiple devices and copying the model on each device. Accelerator provides a convenient interface for data preparation, and you'll learn how to preprocess images, audio, and text as a first step in distributed training.

  • 2

    Distributed Training with Accelerator and Trainer

    In distributed training, each device trains on its data in parallel. You'll investigate two methods for distributed training: Accelerator enables custom training loops, and Trainer simplifies the interface for training.

  • 3

    Improving Training Efficiency

    Distributed training strains resources with large models and datasets, but you can address these challenges by improving memory usage, device communication, and computational efficiency. You'll discover the techniques of gradient accumulation, gradient checkpointing, local stochastic gradient descent, and mixed precision training.

  • 4

    Training with Efficient Optimizers

    You'll focus on optimizers as levers to improve distributed training efficiency, highlighting tradeoffs between AdamW, Adafactor, and 8-bit Adam. Reducing the number of parameters or using low precision helps to decrease a model's memory footprint.

Initializing

Exercise

AdamW with Accelerator

You want to customize your training loop using Accelerator and use AdamW as a benchmark optimizer for your language translation model. Build the training loop to use AdamW.

Some training objects have been pre-loaded and defined, including model, train_dataloader, and accelerator.

Instructions

100 XP
  • Prepare training objects for distributed training before the loop.
  • Update the model parameters in the training loop.