Sentence and word tokenization
Tokenization is an important first step in NLP. It involves breaking text into smaller units called tokens, which is key to working with language data. Your task is to tokenize a snippet of a news article into both sentences and words.
Questo esercizio fa parte del corso
Natural Language Processing (NLP) in Python
Esercizio pratico interattivo
Prova a risolvere questo esercizio completando il codice di esempio.
# Import nltk
____
# Download the punkt_tab package
____
text = """
The stock market saw a significant dip today. Experts believe the downturn may continue.
However, many investors are optimistic about future growth.
"""
# Tokenize the text into sentences
sentences = ____
print(sentences)