Adafactor with Trainer

You're training a Transformer model with billions of parameters for your language translation service. It is straining your computational resources, so you decide to try the Adafactor optimizer to reduce memory requirements compared to AdamW. Prepare the Trainer for Adafactor!

Some training objects have been pre-loaded, including model, train_dataset, validation_dataset, and compute_metrics.

本练习是课程的一部分

Efficient AI Model Training with PyTorch

查看课程

练习说明

Specify Adafactor as an optimizer in TrainingArguments.
Pass in the optimizer state to print the size.

交互式实操练习

通过完成这段示例代码来试试这个练习。

# Specify Adafactor as an optimizer
training_args = TrainingArguments(output_dir="./results",
                                  evaluation_strategy="epoch",
                                  optim="____")

trainer = Trainer(model=model,
                  args=training_args,
                  train_dataset=train_dataset,
                  eval_dataset=validation_dataset,
                  compute_metrics=compute_metrics)
trainer.train()

# Pass in the optimizer state
total_size_megabytes, total_num_elements = compute_optimizer_size(trainer.optimizer.state.____())
print(f"\nNumber of optimizer parameters: {total_num_elements:,}\nOptimizer size: {total_size_megabytes:.0f} MB")

编辑并运行代码