Session Ready
Exercise

Controlling the vocabulary with the Tokenizer

Let's drill down a bit more into the operation of the Tokenizer. In this exercise you will learn how to convert an arbitrary sentence to a sequence using a trained Tokenizer. Furthermore, you will learn to control the size of the vocabulary of the Tokenizer. You will also investigate what happens to the out-of-vocabulary (OOV) words when you limit the vocabulary size of a Tokenizer.

For this exercise, you have been provided with the en_tok Tokenizer that you previously implemented. The Tokenizer has been imported for you.

Instructions
100 XP
  • Convert the following sentence to a sequence using the previous en_tok Tokenizer: she likes grapefruit , peaches , and lemons .
  • Create a new Tokenizer, en_tok_new with a vocabulary size of 50 and out-of-vocabulary word UNK.
  • Fit the new tokenizer on en_text data.
  • Convert the sentence she likes grapefruit , peaches , and lemons . to a sequence with the en_tok_new.