Tokenize a text dataset
You are working on market research for a travel website, and would like to use an existing dataset to fine tune a model that will help you classify hotel reviews. You decide to use the datasets
library.
The AutoTokenizer
class has been pre-imported from transformers
, and load_dataset()
has been pre-imported from datasets
.
This exercise is part of the course
Reinforcement Learning from Human Feedback (RLHF)
Exercise instructions
- Add padding to the tokenizer to process text as equal-sized batches.
- Tokenize the text data using the pre-trained GPT tokenizer and defined function.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
dataset = load_dataset("argilla/tripadvisor-hotel-reviews")
tokenizer = AutoTokenizer.from_pretrained("openai-gpt")
# Add padding with the pad token
tokenizer.____
def tokenize_function(examples):
return tokenizer(examples["text"], padding="max_length", truncation=True)
# Tokenize the dataset
tokenized_datasets = dataset.map(____, batched=True)