Part 1: Exploring the dataset
You will now explore the dataset a little bit. You will first get a feel of what the data looks like. You will print some of the data and learn how to tokenize the sentences in the data to individual words. For the English language, tokenization appears to be a trivial task, however, there are languages such as Japanese, which are not as consistently delimited as English.
For this exercise, you have been provided with two datasets: en_text and fr_text. The en_text contains a list of English sentences, where the fr_text contains the corresponding list of French sentences.
This exercise is part of the course
Machine Translation with Keras
Exercise instructions
- Write a
zip()function that iterates through the first 5 sentences of the English sentences (en_text) and French sentences (fr_text). - Get the first English sentence from
en_text. - Tokenize the obtained sentence using the
split()function and the space character and assign it tofirst_words. - Print the tokenized words.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Iterate through the first 5 English and French sentences in the dataset
for en_sent, fr_sent in zip(____, ____):
print("English: ", en_sent)
print("\tFrench: ", fr_sent)
# Get the first sentence of the English dataset
first_sent = ____[____]
print("First sentence: ", first_sent)
# Tokenize the first sentence
____ = ____.____(____)
# Print the tokenized words
print("\tWords: ", ____)