Get startedGet started for free

Transfer learning using BERT

At PyBooks, the company has decided to leverage the power of the BERT model, a pre-trained transformer model, for sentiment analysis. BERT has seen remarkable performance across various NLP tasks, making it a prime candidate for this use case.

You're tasked with setting up a basic workflow using the BERT model from the transformers library for binary sentiment classification.

The following has been imported for you: BertTokenizer, BertForSequenceClassification, torch. The example data texts and corresponding labels are also preloaded.

This exercise is part of the course

Deep Learning for Text with PyTorch

View Course

Exercise instructions

  • Load the bert-base-uncased tokenizer and model suitable for binary classification.
  • Tokenize your dataset and prepare it for the model, ensuring it returns PyTorch tensors using the return_tensors argument.
  • Setup the optimizer using model parameters.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

# Load the BERT tokenizer and model
tokenizer = BertTokenizer.from_pretrained('____')
model = BertForSequenceClassification.from_pretrained('____', num_labels=____)

# Tokenize your data and return PyTorch tensors
inputs = tokenizer(____, padding=True, truncation=True, return_tensors="____", max_length=32)
inputs["labels"] = torch.tensor(labels)

# Setup the optimizer using model parameters
optimizer = torch.optim.AdamW(____, lr=0.00001)
model.train()
for epoch in range(2):
    outputs = model(**inputs)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    print(f"Epoch: {epoch+1}, Loss: {loss.item()}")
Edit and Run Code