CommencerCommencer gratuitement

Creating input embeddings

Time to begin creating your very own transformer model, and the first step is to embed the input token IDs!

You'll define an InputEmbeddings class with the following parameters:

  • vocab_size: the size of the model vocabulary
  • d_model: the dimensionality of the input embeddings

The torch and math libraries have been imported for you, as well as torch.nn as nn. These will be loaded throughout the course exercises.

Cet exercice fait partie du cours

Transformer Models with PyTorch

Afficher le cours

Instructions

  • Set the model dimensionality and vocabulary size to the d_model and vocab_size arguments, respectively.
  • Instantiate the embedding layer.
  • Return the embeddings multiplied by the square root of self.d_model.
  • Instantiate InputEmbeddings with vocab_size of 10,000 and d_model of 512, and apply it to token_ids.

Exercice interactif pratique

Essayez cet exercice en complétant cet exemple de code.

class InputEmbeddings(nn.Module):
    def __init__(self, vocab_size: int, d_model: int) -> None:
        super().__init__()
        # Set the model dimensionality and vocabulary size
        self.d_model = ____
        self.vocab_size = ____
        # Instantiate the embedding layer
        self.embedding = ____

    def forward(self, x):
        # Return the embeddings multiplied by the square root of d_model
        return ____

# Instantiate InputEmbeddings and apply it to token_ids
embedding_layer = ____
output = ____
print(output.shape)
Modifier et exécuter le code