Transfer learning using BERT
At PyBooks, the company has decided to leverage the power of the BERT model, a pre-trained transformer model, for sentiment analysis. BERT has seen remarkable performance across various NLP tasks, making it a prime candidate for this use case.
You're tasked with setting up a basic workflow using the BERT model from the transformers library for binary sentiment classification.
The following has been imported for you: BertTokenizer
, BertForSequenceClassification
, torch
.
The example data texts
and corresponding labels
are also preloaded.
This exercise is part of the course
Deep Learning for Text with PyTorch
Exercise instructions
- Load the
bert-base-uncased
tokenizer and model suitable for binary classification. - Tokenize your dataset and prepare it for the model, ensuring it returns PyTorch tensors using the
return_tensors
argument. - Setup the optimizer using model parameters.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Load the BERT tokenizer and model
tokenizer = BertTokenizer.from_pretrained('____')
model = BertForSequenceClassification.from_pretrained('____', num_labels=____)
# Tokenize your data and return PyTorch tensors
inputs = tokenizer(____, padding=True, truncation=True, return_tensors="____", max_length=32)
inputs["labels"] = torch.tensor(labels)
# Setup the optimizer using model parameters
optimizer = torch.optim.AdamW(____, lr=0.00001)
model.train()
for epoch in range(2):
outputs = model(**inputs)
loss = outputs.loss
loss.backward()
optimizer.step()
optimizer.zero_grad()
print(f"Epoch: {epoch+1}, Loss: {loss.item()}")