BaşlayınÜcretsiz Başlayın

Tokenizing text with AutoTokenizer

AutoTokenizers simplify text preparation by automatically handling cleaning, normalization, and tokenization. They ensure the text is processed just as the model expects.

In this exercise, explore how AutoTokenizer transforms text into tokens ready for machine learning tasks.

Bu egzersiz

Working with Hugging Face

kursunun bir parçasıdır
Kursu Görüntüle

Egzersiz talimatları

  • Import the required class from transformers, load the tokenizer using the correct method, and split input text into tokens.

Uygulamalı interaktif egzersiz

Bu örnek kodu tamamlayarak bu egzersizi bitirin.

# Import necessary library for tokenization
from transformers import ____

# Load the tokenizer
tokenizer = AutoTokenizer.____("distilbert-base-uncased-finetuned-sst-2-english")

# Split input text into tokens
tokens = tokenizer.____("AI: Making robots smarter and humans lazier!")

# Display the tokenized output
print(f"Tokenized output: {tokens}")
Kodu Düzenle ve Çalıştır