1. Learn
    2. /
  2. Courses
    3. /
  3. Transformer Models with PyTorch
Connected

Exercise

The encoder transformer layer

With a FeedForwardSubLayer class defined, you have all of the pieces you need to define an EncoderLayer class. Recall that the encoder layer typically consists of a multi-head attention mechanism, and a feed-forward sublayer with layer normalization and dropout on the sublayer's inputs and outputs.

The classes you have already defined are available for you with the same names, along with torch and torch.nn as nn.

Instructions

100 XP
  • Complete the __init__ method to instantiate MultiHeadAttention, FeedForwardSubLayer, and two layer normalizations.
  • Complete the forward() method by filling-in the multi-head attention mechanism and feed-forward sublayer; for the attention mechanism, use the src_mark provided and the input embeddings, x, for the query, key, and value matrices.