Get startedGet started for free

Creating input embeddings

Time to begin creating your very own transformer model, and the first step is to embed the input token IDs!

You'll define an InputEmbeddings class with the following parameters:

  • vocab_size: the size of the model vocabulary
  • d_model: the dimensionality of the input embeddings

The torch and math libraries have been imported for you, as well as torch.nn as nn. These will be loaded throughout the course exercises.

This exercise is part of the course

Transformer Models with PyTorch

View Course

Exercise instructions

  • Set the model dimensionality and vocabulary size to the d_model and vocab_size arguments, respectively.
  • Instantiate the embedding layer.
  • Return the embeddings multiplied by the square root of self.d_model.
  • Instantiate InputEmbeddings with vocab_size of 10,000 and d_model of 512, and apply it to token_ids.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

class InputEmbeddings(nn.Module):
    def __init__(self, vocab_size: int, d_model: int) -> None:
        super().__init__()
        # Set the model dimensionality and vocabulary size
        self.d_model = ____
        self.vocab_size = ____
        # Instantiate the embedding layer
        self.embedding = ____

    def forward(self, x):
        # Return the embeddings multiplied by the square root of d_model
        return ____

# Instantiate InputEmbeddings and apply it to token_ids
embedding_layer = ____
output = ____
print(output.shape)
Edit and Run Code