Aan de slagGa gratis aan de slag

Tokenizing text with AutoTokenizer

AutoTokenizers simplify text preparation by automatically handling cleaning, normalization, and tokenization. They ensure the text is processed just as the model expects.

In this exercise, explore how AutoTokenizer transforms text into tokens ready for machine learning tasks.

Deze oefening maakt deel uit van de cursus

Working with Hugging Face

Cursus bekijken

Oefeninstructies

  • Import the required class from transformers, load the tokenizer using the correct method, and split input text into tokens.

Praktische interactieve oefening

Probeer deze oefening eens door deze voorbeeldcode in te vullen.

# Import necessary library for tokenization
from transformers import ____

# Load the tokenizer
tokenizer = AutoTokenizer.____("distilbert-base-uncased-finetuned-sst-2-english")

# Split input text into tokens
tokens = tokenizer.____("AI: Making robots smarter and humans lazier!")

# Display the tokenized output
print(f"Tokenized output: {tokens}")
Code bewerken en uitvoeren