IniziaInizia gratis

Tokenizing text with AutoTokenizer

AutoTokenizers simplify text preparation by automatically handling cleaning, normalization, and tokenization. They ensure the text is processed just as the model expects.

In this exercise, explore how AutoTokenizer transforms text into tokens ready for machine learning tasks.

Questo esercizio fa parte del corso

Working with Hugging Face

Visualizza il corso

Istruzioni dell'esercizio

  • Import the required class from transformers, load the tokenizer using the correct method, and split input text into tokens.

Esercizio pratico interattivo

Prova a risolvere questo esercizio completando il codice di esempio.

# Import necessary library for tokenization
from transformers import ____

# Load the tokenizer
tokenizer = AutoTokenizer.____("distilbert-base-uncased-finetuned-sst-2-english")

# Split input text into tokens
tokens = tokenizer.____("AI: Making robots smarter and humans lazier!")

# Display the tokenized output
print(f"Tokenized output: {tokens}")
Modifica ed esegui il codice