Adding special tokens

You will now learn to add sos (marks the start) and eos (marks the end) tokens to the sentences. As already discussed, this step is optional for the model you have right now, but these will be required for a model that you'll be implementing in a later chapter.

To add these special tokens, you will use the Python string.join() function. string.join() joins a list of strings to a single string using a delimiter. For example, if you want to convert ['datacamp', 'is', 'awesome'] to 'datacamp is awesome', you can use " ".join(['datacamp', 'is', 'awesome']), where the " " (i.e. space character) is the delimiter.

For this exercise, a small sample of 10 French sentences has already been imported.

Questo esercizio fa parte del corso

Machine Translation with Keras

Visualizza il corso

Istruzioni dell'esercizio

Loop through the list of French sentences (fr_text).
Add a "sos" token to denote the beginning and an "eos" token to denote the ending of each sentence using the string.join() function.
Append the modified sentence to fr_text_new.
Print the modified sentence sent_new.

Esercizio pratico interattivo

Prova a risolvere questo esercizio completando il codice di esempio.

fr_text_new = []

# Loop through all sentences in fr_text
for sent in ____:
  
  print("Before adding tokens: ", sent)
  
  # Add sos and eos tokens using string.join
  sent_new = " ".____([____, sent, ____])
  # Append the modified sentence to fr_text_new
  ____.____(____)
  
  # Print sentence after adding tokens
  print("After adding tokens: ", ____, '\n')

Modifica ed esegui il codice