AdamW with Accelerator

You want to customize your training loop using Accelerator and use AdamW as a benchmark optimizer for your language translation model. Build the training loop to use AdamW.

Some training objects have been pre-loaded and defined, including model, train_dataloader, and accelerator.

本练习是课程的一部分

Efficient AI Model Training with PyTorch

查看课程

练习说明

Prepare training objects for distributed training before the loop.
Update the model parameters in the training loop.

交互式实操练习

通过完成这段示例代码来试试这个练习。

optimizer = AdamW(params=model.parameters())

# Prepare training objects for distributed training
model, optimizer, train_dataloader = ____.____(model, optimizer, train_dataloader)

for batch in train_dataloader:
    inputs, targets = batch["input_ids"], batch["labels"]
    outputs = model(inputs, labels=targets)
    loss = outputs.loss
    accelerator.backward(loss)
    # Update the model parameters
    ____.____()
    optimizer.zero_grad()
    print(f"Loss = {loss}")

编辑并运行代码