Training loops before and after Accelerator

You want to modify a PyTorch training loop to use Accelerator for your language model to simplify translations using the MPRC dataset of sentence paraphrases. Update the training loop to prepare your model for distributed training.

Some data has been pre-loaded:

accelerator is an instance of Accelerator
train_dataloader, optimizer, model, and lr_scheduler have been defined and prepared with Accelerator

本练习是课程的一部分

Efficient AI Model Training with PyTorch

查看课程

练习说明

Update the .to(device) lines so that Accelerator handles device placement.
Modify the gradient computation to use Accelerator.

交互式实操练习

通过完成这段示例代码来试试这个练习。

for batch in train_dataloader:
    optimizer.zero_grad()
    inputs, targets = batch["input_ids"], batch["labels"]
    # Update the lines so Accelerator handles device placement
    inputs = inputs.to(device)
    targets = targets.to(device)
    outputs = model(inputs, labels=targets)
    loss = outputs.loss
    # Modify the gradient computation to use Accelerator
    ____.backward(____)
    optimizer.step()
    lr_scheduler.step()

编辑并运行代码