Save and load the state of preprocessed text

You've finished preprocessing a dataset of agricultural questions and answers to help farmers look up common issues in the field. To save your progress, you decide to create and load checkpoints of the preprocessed text data.

Some data has been preloaded:

dataloader contains a preprocessed dataset of agricultural questions and answers
Accelerator has been imported from accelerate

本练习是课程的一部分

Efficient AI Model Training with PyTorch

查看课程

练习说明

Save a checkpoint of the preprocessed data in checkpoint_dir.
Load a checkpoint from checkpoint_dir.

交互式实操练习

通过完成这段示例代码来试试这个练习。

accelerator = Accelerator()

dataloader = accelerator.prepare(dataloader)

checkpoint_dir = Path("preprocess_checkpoint")

# Save a checkpoint of the data
____.____(checkpoint_dir)

# Load from the checkpoint
____.____(checkpoint_dir)

编辑并运行代码