1. Learn
    2. /
  2. Courses
    3. /
  3. Multi-Modal Models with Hugging Face
Connected

Exercise

Model evaluation on a custom dataset

In this exercise, you will use an evaluator from the Hugging Face evaluate package to assess the performance of a pretrained model on a custom dataset. Note that, for multi-class classification with dataset imbalances, accuracy is not a reliable performance indicator. Therefore, you will use the ability of the evaluator to provide multiple measures at once: the precision and recall.

A dataset (dataset) and pipeline (pipe) have been predefined. The evaluate library and the evaluator class have also already been imported.

Instructions

100 XP
  • Instantiate an evaluator for your "image-classification" task.
  • Extract the integer-to-string label mapping from the pipeline.
  • Evaluate the dataset (dataset) and pipeline (pipe) using the metrics stored in metrics_dict and label_map.