Sentence and word tokenization
Tokenization is an important first step in NLP. It involves breaking text into smaller units called tokens, which is key to working with language data. Your task is to tokenize a snippet of a news article into both sentences and words.
Latihan ini adalah bagian dari kursus
Natural Language Processing (NLP) in Python
Latihan interaktif praktis
Cobalah latihan ini dengan menyelesaikan kode contoh berikut.
# Import nltk
____
# Download the punkt_tab package
____
text = """
The stock market saw a significant dip today. Experts believe the downturn may continue.
However, many investors are optimistic about future growth.
"""
# Tokenize the text into sentences
sentences = ____
print(sentences)