Adding the transformer head
Time to design a transformer head that could be used for classification tasks like sentiment analysis or categorization. You'll define a ClassifierHead class, create instances of the body and head, and pass a series of token IDs through them both to test that they work as expected.
Note: because this model has been trained yet, the outputs will be meaningless, but testing the code can process inputs and generate outputs in the form you expect is a good test.
This exercise is part of the course
Transformer Models with PyTorch
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Complete the classification head
class ClassifierHead(nn.Module):
def __init__(self, d_model, num_classes):
super().__init__()
self.fc = ____
def forward(self, x):
logits = self.fc(x)
return F.____(logits, dim=-1)