Tokenizing text with AutoTokenizer
AutoTokenizers simplify text preparation by automatically handling cleaning, normalization, and tokenization. They ensure the text is processed just as the model expects.
In this exercise, explore how AutoTokenizer transforms text into tokens ready for machine learning tasks.
This exercise is part of the course
Working with Hugging Face
Exercise instructions
- Import the required class from
transformers, load the tokenizer using the correct method, and split input text into tokens.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Import necessary library for tokenization
from transformers import ____
# Load the tokenizer
tokenizer = AutoTokenizer.____("distilbert-base-uncased-finetuned-sst-2-english")
# Split input text into tokens
tokens = tokenizer.____("AI: Making robots smarter and humans lazier!")
# Display the tokenized output
print(f"Tokenized output: {tokens}")