CV fine-tuning: model classes

In this exercise, you will load the pretrained model, and adapt the output to accommodate a new classification of car model types from the Stanford Cars dataset instead of the 1000 classes used for the original ImageNet training. The dataset contains labelled images of cars.

The dataset has been loaded (dataset), as has AutoModelForImageClassification from transformers. The dataset has been filtered so that three model types are included.

This exercise is part of the course

Multi-Modal Models with Hugging Face

Exercise instructions

Obtain the new label names from the dataset
Add the new id2label mapping while loading the model.
Add the corresponding label2id mapping.
Add the required flag to change the number of classes.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

# Obtain the new label names from the dataset
labels = dataset["train"].features["____"].____

label2id, id2label = dict(), dict()
for i, label in enumerate(labels):
    label2id[label] = str(i)
    id2label[str(i)] = label

model = AutoModelForImageClassification.from_pretrained(
    "google/mobilenet_v2_1.0_224",
    num_labels=len(labels),
    # Add the id2label mapping
    id2label=____,
    # Add the corresponding label2id mapping
    label2id=____,
    # Add the required flag to change the number of classes
    ignore_mismatched_sizes=____
)

Edit and Run Code

This exercise is part of the course

Multi-Modal Models with Hugging Face

IntermediateSkill Level

5.0+

Start Course for Free

Navigate the Hugging Face model hub, transform raw text, audio, and visual data into AI-friendly formats. Learn how to find the latest most popular models for tasks such as text generation and harness the power of pre-built pipelines.

Exercise 1: Hugging Face model navigation Exercise 2: How many models!?Exercise 3: Finding the most popular text-to-image model Exercise 4: Loading and preprocessing data Exercise 5: Text tokenizing Exercise 6: Image preprocessing Exercise 7: Audio preprocessing Exercise 8: Pipeline tasks and evaluations Exercise 9: Pipeline caption generation Exercise 10: Passing keyword arguments Exercise 11: Model evaluation on a custom dataset

Learn to master individual modalities with state-of-the-art models. Dive into computer vision for image classification and segmentation, explore speech recognition and text-to-speech synthesis, and learn effective fine-tuning techniques. Build practical skills with pre-trained models from Hugging Face's transformers library.

Exercise 1: Computer vision Exercise 2: Image classification Exercise 3: Object detection Exercise 4: Image background removal Exercise 5: Fine-tuning computer vision models Exercise 6: CV fine-tuning: dataset prep Exercise 7: CV fine-tuning: model classes

Current Exercise

Exercise 8: CV fine-tuning: trainer configuration Exercise 9: Speech recognition and audio generation Exercise 10: Automatic speech recognition Exercise 11: Creating speech embeddings Exercise 12: Audio denoising Exercise 13: Fine-tuning text-to-speech models Exercise 14: Fine-tuning a text-to-speech model Exercise 15: Generating new speech

Learn to fuse visual, textual, and audio information for richer AI applications. Master techniques like CLIP for zero-shot classification, build sentiment analyzers that see and read, and create emotion detectors that combine facial expressions with voice. Take your AI models beyond single-modality thinking.

Exercise 1: Image-text similarity Exercise 2: Zero-shot learning with CLIP Exercise 3: Automated caption quality assessment Exercise 4: Multimodal sentiment analysis Exercise 5: Vision Language models: multi-modal setup Exercise 6: Vision Language models: multi-modal sentiment Exercise 7: Audio-visual emotion analysis Exercise 8: Video audio splitting Exercise 9: CLIP CLAP: Multi-modal emotion classification

Transform ideas into reality! Master cutting-edge AI techniques to generate and manipulate visual content using text prompts. Create stunning images, edit photos intelligently, and build powerful question-answering systems for images and documents. Turn your creative vision into digital reality with multi-modal AI.

Exercise 1: Multimodal QA tasks Exercise 2: Image-text to text with ViLT Exercise 3: Document-text to text with LayoutLM Exercise 4: Text-guided image editing Exercise 5: Custom image editing Exercise 6: Image inpainting Exercise 7: Multimodal video generation Exercise 8: Video generation Exercise 9: Assessing video generation performance Exercise 10: Congratulations!