1. Learn
    2. /
  2. Courses
    3. /
  3. Introduction to LLMs in Python
Connected

Exercise

BLEU translations

Let's get familiar with the BLEU metric.

A pipeline based on the Helsinki-NLP Spanish-English translation model and the BLEU metric has been loaded for you, using evaluate.load("bleu") from the evaluate library.

Given the following inputs and references for evaluation:

input_sentence_1 = "Hola, ¿cómo estás?"

reference_1 = [
     ["Hello, how are you?", "Hi, how are you?"]
     ]

input_sentences_2 = ["Hola, ¿cómo estás?", "Estoy genial, gracias."]

references_2 = [
     ["Hello, how are you?", "Hi, how are you?"],
     ["I'm great, thanks.", "I'm great, thank you."]
     ]

Instructions 1/2

undefined XP
  • 1
    • Pass the input sentence in input_sentence_1 to the translator, then calculate the BLEU metric using reference_1.
  • 2
    • Repeat the process, passing in the input_sentences_2 list containing two Spanish sentences to translate.