Creating input embeddings
Time to begin creating your very own transformer model, and the first step is to embed the input token IDs!
You'll define an InputEmbeddings
class with the following parameters:
vocab_size
: the size of the model vocabularyd_model
: the dimensionality of the input embeddings
The torch
and math
libraries have been imported for you, as well as torch.nn
as nn
. These will be loaded throughout the course exercises.
This exercise is part of the course
Transformer Models with PyTorch
Exercise instructions
- Set the model dimensionality and vocabulary size to the
d_model
andvocab_size
arguments, respectively. - Instantiate the embedding layer.
- Return the embeddings multiplied by the square root of
self.d_model
. - Instantiate
InputEmbeddings
withvocab_size
of 10,000 andd_model
of 512, and apply it totoken_ids
.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
class InputEmbeddings(nn.Module):
def __init__(self, vocab_size: int, d_model: int) -> None:
super().__init__()
# Set the model dimensionality and vocabulary size
self.d_model = ____
self.vocab_size = ____
# Instantiate the embedding layer
self.embedding = ____
def forward(self, x):
# Return the embeddings multiplied by the square root of d_model
return ____
# Instantiate InputEmbeddings and apply it to token_ids
embedding_layer = ____
output = ____
print(output.shape)