BaşlayınÜcretsiz Başlayın

The encoder transformer layer

With a FeedForwardSubLayer class defined, you have all of the pieces you need to define an EncoderLayer class. Recall that the encoder layer typically consists of a multi-head attention mechanism, and a feed-forward sublayer with layer normalization and dropout on the sublayer's inputs and outputs.

The classes you have already defined are available for you with the same names, along with torch and torch.nn as nn.

Bu egzersiz

Transformer Models with PyTorch

kursunun bir parçasıdır
Kursu Görüntüle

Egzersiz talimatları

  • Complete the __init__ method to instantiate MultiHeadAttention, FeedForwardSubLayer, and two layer normalizations.
  • Complete the forward() method by filling-in the multi-head attention mechanism and feed-forward sublayer; for the attention mechanism, use the src_mark provided and the input embeddings, x, for the query, key, and value matrices.

Uygulamalı interaktif egzersiz

Bu örnek kodu tamamlayarak bu egzersizi bitirin.

class EncoderLayer(nn.Module):
    def __init__(self, d_model, num_heads, d_ff, dropout):
        super().__init__()
        # Instantiate the layers
        self.self_attn = ____(d_model, num_heads)
        self.ff_sublayer = ____(d_model, d_ff)
        self.norm1 = ____
        self.norm2 = ____
        self.dropout = nn.Dropout(dropout)

    def forward(self, x, src_mask):
        # Complete the forward method
        attn_output = self.____
        x = self.norm1(x + self.dropout(attn_output))
        ff_output = self.____
        x = self.norm2(x + self.dropout(ff_output))
        return x
Kodu Düzenle ve Çalıştır