MulaiMulai sekarang secara gratis

Mapping tokenization

You now want to test out having more control over the tokenization and want to try tokenizing the data in rows or batches. This will also give you a result that is a DataSet object, which you'll need for training.

The tokenizer has been loaded for you along with the data as train_data and test_data.

Latihan ini adalah bagian dari kursus

Introduction to LLMs in Python

Lihat Kursus

Latihan interaktif praktis

Cobalah latihan ini dengan menyelesaikan kode contoh berikut.

# Complete the function
def tokenize_function(data):
    return tokenizer(data["interaction"], 
                     ____, 
                     padding=True, 
                     ____, 
                     max_length=64)

tokenized_in_batches = ____
Edit dan Jalankan Kode