1. Učit se
    2. /
  2. Kurzy
    3. /
  3. Efficient AI Model Training with PyTorch
Connected

cvičení

Mixed precision training with basic PyTorch

You will use low precision floating point data types to speed up training for your language translation model. For example, 16-bit floating point data types (float16) are only half the size of their 32-bit counterparts (float32). This accelerates computations of matrix multiplications and convolutions. Recall that this involves scaling gradients and casting operations to 16 bit floating point.

Some objects have been preloaded: dataset, model, dataloader, and optimizer.

Pokyny

100 XP
  • Before the loop, define a scaler for the gradients using torch.amp.GradScaler.
  • In the loop, cast operations to the 16-bit floating point data type using torch.autocast as a context manager.
  • In the loop, scale the loss and call .backward() to create scaled gradients.