Adafactor with Trainer

You're training a Transformer model with billions of parameters for your language translation service. It is straining your computational resources, so you decide to try the Adafactor optimizer to reduce memory requirements compared to AdamW. Prepare the Trainer for Adafactor!

Some training objects have been pre-loaded, including model, train_dataset, validation_dataset, and compute_metrics.

Это упражнение является частью курса

Efficient AI Model Training with PyTorch

Посмотреть курс

Инструкции к упражнению

Specify Adafactor as an optimizer in TrainingArguments.
Pass in the optimizer state to print the size.

Интерактивное практическое упражнение

Попробуйте выполнить это упражнение, дополнив этот пример кода.

# Specify Adafactor as an optimizer
training_args = TrainingArguments(output_dir="./results",
                                  evaluation_strategy="epoch",
                                  optim="____")

trainer = Trainer(model=model,
                  args=training_args,
                  train_dataset=train_dataset,
                  eval_dataset=validation_dataset,
                  compute_metrics=compute_metrics)
trainer.train()

# Pass in the optimizer state
total_size_megabytes, total_num_elements = compute_optimizer_size(trainer.optimizer.state.____())
print(f"\nNumber of optimizer parameters: {total_num_elements:,}\nOptimizer size: {total_size_megabytes:.0f} MB")

Редактировать и запускать код