1. Learn
    2. /
  2. Courses
    3. /
  3. Transformer Models with PyTorch
  • 1

    The Building Blocks of Transformer Models

    Free

    Discover what makes the hottest deep learning architecture in AI tick! Learn about the components that make up Transformer models, including the famous self-attention mechanisms described in the renowned paper "Attention is All You Need."

  • 2

    Building Transformer Architectures

    Design transformer encoder and decoder blocks, and combine them with positional encoding, multi-headed attention, and position-wise feed-forward networks to build your very own Transformer architectures. Along the way, you'll develop a deep understanding and appreciation for how transformers work under the hood.

Initializing

Exercise

Starting the MultiHeadAttentionClass

Now that you've defined classes for creating token embeddings and positional embeddings, it's time to define a class for performing multi-head attention. To start, set up the parameters used for the attention calculation and the linear layers used for transforming the input embeddings into query, key, and value matrices, and one for projecting the combined attention weights back into embeddings.

torch.nn has been imported as nn.

Instructions

100 XP
  • Calculate the embedding dimensions each attention head will process, head_dim.
  • Define the three input layers (for query, key, and value) and one output layer; remove the bias parameter from the input layers.