ComeçarComece de graça

Creating input embeddings

Time to begin creating your very own transformer model, and the first step is to embed the input token IDs!

You'll define an InputEmbeddings class with the following parameters:

  • vocab_size: the size of the model vocabulary
  • d_model: the dimensionality of the input embeddings

The torch and math libraries have been imported for you, as well as torch.nn as nn. These will be loaded throughout the course exercises.

Este exercício faz parte do curso

Transformer Models with PyTorch

Ver curso

Instruções do exercício

  • Set the model dimensionality and vocabulary size to the d_model and vocab_size arguments, respectively.
  • Instantiate the embedding layer.
  • Return the embeddings multiplied by the square root of self.d_model.
  • Instantiate InputEmbeddings with vocab_size of 10,000 and d_model of 512, and apply it to token_ids.

Exercício interativo prático

Experimente este exercício completando este código de exemplo.

class InputEmbeddings(nn.Module):
    def __init__(self, vocab_size: int, d_model: int) -> None:
        super().__init__()
        # Set the model dimensionality and vocabulary size
        self.d_model = ____
        self.vocab_size = ____
        # Instantiate the embedding layer
        self.embedding = ____

    def forward(self, x):
        # Return the embeddings multiplied by the square root of d_model
        return ____

# Instantiate InputEmbeddings and apply it to token_ids
embedding_layer = ____
output = ____
print(output.shape)
Editar e executar o código