1. Learn
    2. /
  2. Courses
    3. /
  3. Transformer Models with PyTorch
Connected

Exercise

Creating positional encodings

Embedding the tokens is a good start, but these embeddings still lack information about each token's position in the sequence. To remedy this, the transformer architecture makes use of positional encodings. This encodes positional information from each token into the embeddings.

You'll create a PositionalEncoding class with the following parameters:

  • d_model: the dimensionality of the input embeddings
  • max_seq_length: the maximum sequence length (or the sequence length if each sequence is the same length)

Instructions

100 XP
  • Create a matrix of zeros of dimensions max_seq_length by d_model.
  • Perform the sine and cosine calculations on position * div_term to create the even and odd positional embedding values.
  • Ensure pe isn't a learnable parameter during training.
  • Add the transformed positional embeddings to the input token embeddings, x.