1. Học hỏi
    2. /
  2. Khoa Học
    3. /
  3. Efficient AI Model Training with PyTorch
Connected

Bài tập

Local SGD with Accelerator

You've implemented gradient accumulation and gradient checkpointing to streamline memory usage for your language translation model. Training is still a bit slow, so you decide to add local SGD to your training loop to improve communication efficiency between devices. Build the training loop with local SGD!

The model, train_dataloader, and accelerator have been pre-defined, and LocalSGD has been imported.

Hướng dẫn

100 XP
  • Set local_sgd_steps to synchronize gradients every eight steps.
  • Step the local SGD context manager.