Tokenizing text with AutoTokenizer
AutoTokenizers simplify text preparation by automatically handling cleaning, normalization, and tokenization. They ensure the text is processed just as the model expects.
In this exercise, explore how AutoTokenizer transforms text into tokens ready for machine learning tasks.
Latihan ini adalah bagian dari kursus
Working with Hugging Face
Petunjuk latihan
- Import the required class from
transformers, load the tokenizer using the correct method, and split input text into tokens.
Latihan interaktif praktis
Cobalah latihan ini dengan menyelesaikan kode contoh berikut.
# Import necessary library for tokenization
from transformers import ____
# Load the tokenizer
tokenizer = AutoTokenizer.____("distilbert-base-uncased-finetuned-sst-2-english")
# Split input text into tokens
tokens = tokenizer.____("AI: Making robots smarter and humans lazier!")
# Display the tokenized output
print(f"Tokenized output: {tokens}")