LoslegenKostenlos loslegen

Completing the decoder transformer

Time to build the decoder transformer body! This will mean combining the InputEmbeddings, PositionalEncoding, and DecoderLayer classes you've created previously.

Diese Übung ist Teil des Kurses

Transformer Models with PyTorch

Kurs anzeigen

Anleitung zur Übung

  • Define a list of num_layers decoder layers using a list comprehension and the DecoderLayer class.
  • Define a linear layer to project the hidden states into word likelihoods.
  • Complete the forward pass through the layers defined in __init__.
  • Instantiate a decoder transformer and apply it to input_tokens and tgt_mask.

Interaktive Übung

Versuche dich an dieser Übung, indem du diesen Beispielcode vervollständigst.

class TransformerDecoder(nn.Module):
    def __init__(self, vocab_size, d_model, num_layers, num_heads, d_ff, dropout, max_seq_length):
        super(TransformerDecoder, self).__init__()
        self.embedding = InputEmbeddings(vocab_size, d_model)
        self.positional_encoding = PositionalEncoding(d_model, max_seq_length)
        # Define the list of decoder layers and linear layer
        self.layers = nn.____([____(d_model, num_heads, d_ff, dropout) for _ in range(num_layers)])
        # Define a linear layer to project hidden states to likelihoods
        self.fc = ____
  
    def forward(self, x, tgt_mask):
        # Complete the forward pass
        x = self.____(x)
        x = self.____(x)
        for layer in self.layers:
            x = ____
        x = self.____(x)
        return F.log_softmax(x, dim=-1)

# Instantiate a decoder transformer and apply it to input_tokens and tgt_mask
transformer_decoder = ____(vocab_size, d_model, num_layers, num_heads, d_ff, dropout, max_seq_length)   
output = ____
print(output)
print(output.shape)
Code bearbeiten und ausführen