Sentence and word tokenization
Tokenization is an important first step in NLP. It involves breaking text into smaller units called tokens, which is key to working with language data. Your task is to tokenize a snippet of a news article into both sentences and words.
Deze oefening maakt deel uit van de cursus
Natural Language Processing (NLP) in Python
Praktische interactieve oefening
Probeer deze oefening eens door deze voorbeeldcode in te vullen.
# Import nltk
____
# Download the punkt_tab package
____
text = """
The stock market saw a significant dip today. Experts believe the downturn may continue.
However, many investors are optimistic about future growth.
"""
# Tokenize the text into sentences
sentences = ____
print(sentences)