1. Apprendre
    2. /
  2. Cours
    3. /
  3. Transformer Models with PyTorch
Connected

Exercice

Creating input embeddings

Time to begin creating your very own transformer model, and the first step is to embed the input token IDs!

You'll define an InputEmbeddings class with the following parameters:

  • vocab_size: the size of the model vocabulary
  • d_model: the dimensionality of the input embeddings

The torch and math libraries have been imported for you, as well as torch.nn as nn. These will be loaded throughout the course exercises.

Instructions

100 XP
  • Set the model dimensionality and vocabulary size to the d_model and vocab_size arguments, respectively.
  • Instantiate the embedding layer.
  • Return the embeddings multiplied by the square root of self.d_model.
  • Instantiate InputEmbeddings with vocab_size of 10,000 and d_model of 512, and apply it to token_ids.