CLIP CLAP: Multi-modal emotion classification

Now you'll perform the emotion analysis of the advertisement you previously prepared using CLIP/CLAP. In order to make a multi-modal classification of emotion, you will be combining the predictions of these models using the mean (known as late fusion).

The video (video) and the corresponding audio (audio_sample) you created previously is still available:

Frames from the Bounce TV commercial

A list of emotions has been loaded as emotions.

This exercise is part of the course

Multi-Modal Models with Hugging Face

Exercise instructions

Make an audio classifier pipeline for zero-shot-audio-classification using the laion/clap-htsat-unfused model.
Make an image classifier pipeline for zero-shot-image-classification using the openai/clip-vit-large-patch14 model.
Use the image classifier pipeline to generate predictions for each image in the video.
Use the audio classifier pipeline to generate predictions for the audio_sample.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

# Make an audio classifier pipeline
audio_classifier = ____(model=____, task=____)

# Make an image classifier pipeline
image_classifier = ____(model=____, task=____)

scores = []
for img in video:
	# Make image scores
    predictions = ____(____, candidate_labels=____)
    scores.append({l['label']: l['score'] for l in predictions})

av_scores = {emotion: sum([s[emotion] for s in scores])/len(scores) for emotion in emotions}

# Make audio scores
audio_scores = ____(____, candidate_labels=____)

audio_scores = {l['label']: l['score'] for l in audio_scores}
multimodal_scores = {emotion: (av_scores[emotion] + audio_scores[emotion])/2 for emotion in emotions}
print(f"Multimodal scores: {multimodal_scores}")

Edit and Run Code

This exercise is part of the course

Multi-Modal Models with Hugging Face

IntermediateSkill Level

5.0+

Start Course for Free

Navigate the Hugging Face model hub, transform raw text, audio, and visual data into AI-friendly formats. Learn how to find the latest most popular models for tasks such as text generation and harness the power of pre-built pipelines.

Exercise 1: Hugging Face model navigation Exercise 2: How many models!?Exercise 3: Finding the most popular text-to-image model Exercise 4: Loading and preprocessing data Exercise 5: Text tokenizing Exercise 6: Image preprocessing Exercise 7: Audio preprocessing Exercise 8: Pipeline tasks and evaluations Exercise 9: Pipeline caption generation Exercise 10: Passing keyword arguments Exercise 11: Model evaluation on a custom dataset

Learn to master individual modalities with state-of-the-art models. Dive into computer vision for image classification and segmentation, explore speech recognition and text-to-speech synthesis, and learn effective fine-tuning techniques. Build practical skills with pre-trained models from Hugging Face's transformers library.

Exercise 1: Computer vision Exercise 2: Image classification Exercise 3: Object detection Exercise 4: Image background removal Exercise 5: Fine-tuning computer vision models Exercise 6: CV fine-tuning: dataset prep Exercise 7: CV fine-tuning: model classes Exercise 8: CV fine-tuning: trainer configuration Exercise 9: Speech recognition and audio generation Exercise 10: Automatic speech recognition Exercise 11: Creating speech embeddings Exercise 12: Audio denoising Exercise 13: Fine-tuning text-to-speech models Exercise 14: Fine-tuning a text-to-speech model Exercise 15: Generating new speech

Learn to fuse visual, textual, and audio information for richer AI applications. Master techniques like CLIP for zero-shot classification, build sentiment analyzers that see and read, and create emotion detectors that combine facial expressions with voice. Take your AI models beyond single-modality thinking.

Exercise 1: Image-text similarity Exercise 2: Zero-shot learning with CLIP Exercise 3: Automated caption quality assessment Exercise 4: Multimodal sentiment analysis Exercise 5: Vision Language models: multi-modal setup Exercise 6: Vision Language models: multi-modal sentiment Exercise 7: Audio-visual emotion analysis Exercise 8: Video audio splitting Exercise 9: CLIP CLAP: Multi-modal emotion classification

Current Exercise

Transform ideas into reality! Master cutting-edge AI techniques to generate and manipulate visual content using text prompts. Create stunning images, edit photos intelligently, and build powerful question-answering systems for images and documents. Turn your creative vision into digital reality with multi-modal AI.

Exercise 1: Multimodal QA tasks Exercise 2: Image-text to text with ViLT Exercise 3: Document-text to text with LayoutLM Exercise 4: Text-guided image editing Exercise 5: Custom image editing Exercise 6: Image inpainting Exercise 7: Multimodal video generation Exercise 8: Video generation Exercise 9: Assessing video generation performance Exercise 10: Congratulations!