Preprocess text with AutoTokenizer

You're building a precision agriculture application to enable farmers to ask questions on issues they encounter in the field. You'll leverage a dataset of common questions and answers to issues faced by farmers; the fields in this dataset are

question: common agricultural questions
answers: answers to the agricultural questions

As a first step in distributed training, you'll begin by preprocessing this text dataset.

Some data has been preloaded:

dataset contains a sample dataset of agricultural questions and answers
AutoTokenizer has been imported from transformers

本练习是课程的一部分

Efficient AI Model Training with PyTorch

查看课程

练习说明

Load a pre-trained tokenizer.
Tokenize example["question"] using the tokenizer.
Apply the encode() function to the dataset.

交互式实操练习

通过完成这段示例代码来试试这个练习。

# Load a pre-trained tokenizer
tokenizer = ____.____("distilbert-base-uncased")

def encode(example):
    # Tokenize the "question" field of the training example
    return ____(____["____"], padding="max_length", truncation=True, return_tensors="pt")

# Map the function to the dataset
dataset = ____.____(____, batched=True)

dataset = dataset.map(lambda example: {"labels": example["answers"]}, batched=True)

print(dataset)

编辑并运行代码