Get startedGet started for free

Defining the embedding model

You will be defining a Keras model that:

  • Uses Embedding layers
  • Will be trained with Teacher Forcing

This model will have two embedding layers; an encoder embedding layer and a decoder embedding layer. Furthermore, as the model is trained using Teacher Forcing, it will use a sequence length of fr_len-1 in the decoder Input layer.

For this exercise, you have all the required keras.layers and Model imported. Furthermore the variables, en_len (English sequence length), fr_len (French sequence length), en_vocab (English vocabulary size), fr_vocab (French vocabulary size) and hsize (hidden size) have been defined.

This exercise is part of the course

Machine Translation with Keras

View Course

Exercise instructions

  • Define an Input layer which accepts a sequence of word IDs.
  • Define an Embedding layer that embeds en_vocab words, has length 96 and can accept a sequence of IDs (sequence length is specified using the input_length argument).
  • Define an Embedding layer that embeds fr_vocab words, has length 96 and can accept a sequence of fr_len-1 IDs.
  • Define a model that takes an input from the encoder and an input from the decoder (in that order) and outputs the word predictions.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

# Define an input layer which accepts a sequence of word IDs
en_inputs = Input(____=(____,))
# Define an Embedding layer which accepts en_inputs
en_emb = ____(____, ____, input_length=____)(en_inputs)
en_out, en_state = GRU(hsize, return_state=True)(en_emb)

de_inputs = Input(shape=(fr_len-1,))
# Define an Embedding layer which accepts de_inputs
de_emb = Embedding(____, 96, input_length=____)(____)
de_out, _ = GRU(hsize, return_sequences=True, return_state=True)(de_emb, initial_state=en_state)
de_pred = TimeDistributed(Dense(fr_vocab, activation='softmax'))(de_out)

# Define the Model which accepts encoder/decoder inputs and outputs predictions 
nmt_emb = Model([____, ____], ____)
nmt_emb.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['acc'])
Edit and Run Code