Save and load the state of preprocessed text
You've finished preprocessing a dataset of agricultural questions and answers to help farmers look up common issues in the field. To save your progress, you decide to create and load checkpoints of the preprocessed text data.
Some data has been preloaded:
dataloadercontains a preprocessed dataset of agricultural questions and answersAcceleratorhas been imported fromaccelerate
Questo esercizio fa parte del corso
Efficient AI Model Training with PyTorch
Istruzioni dell'esercizio
- Save a checkpoint of the preprocessed data in
checkpoint_dir. - Load a checkpoint from
checkpoint_dir.
Esercizio pratico interattivo
Prova a risolvere questo esercizio completando il codice di esempio.
accelerator = Accelerator()
dataloader = accelerator.prepare(dataloader)
checkpoint_dir = Path("preprocess_checkpoint")
# Save a checkpoint of the data
____.____(checkpoint_dir)
# Load from the checkpoint
____.____(checkpoint_dir)