Automatisierte Qualitätsbewertung von Bildunterschriften

Du hast das Bild des Kleids korrekt klassifiziert, aber wie gut war die ursprüngliche Beschreibung?

Maa Fab wrap with a Trendy design dress with Vibrant color for an elegant touch of Fabric completely Soft and Comfortable wear with amazing design of Solid Boat ? Neck Flared Dress to make a perfect addition to your wardrobe collection.

Jetzt verwendest du das CLIP-Modell, um mithilfe des CLIP-Scores quantitativ zu beurteilen, wie treffend diese Beschreibung ist. Die Bildunterschrift (description), das Bild (image), die Klasse ToTensor und die Funktion clip_score() aus torchmetrics wurden geladen.

Diese Übung ist Teil des Kurses

Multimodale Modelle mit Hugging Face

Anleitung zur Übung

Wandle das Bild in einen PyTorch-Tensor mit Intensitäten im Bereich 0–255 um.
Verwende die Funktion clip_score(), um die Qualität der Bildunterschrift zu bewerten, indem du image_tensor und description mit dem Modell openai/clip-vit-base-patch32 vergleichst.

Interaktive Übung

Vervollständige den Beispielcode, um diese Übung erfolgreich abzuschließen.

# Convert the image to a PyTorch tensor
image_tensor = ____()(image)*____

# Use the clip_score function to assess the quality of the caption
score = ____(____, ____, "____")

print(f"CLIP score: {score}")

Code bearbeiten und ausführen

Diese Übung ist Teil des Kurses

Multimodale Modelle mit Hugging Face

Mittlere SchwierigkeitSchwierigkeitsgrad

4.8+

Kurs kostenlos starten

Navigate the Hugging Face model hub, transform raw text, audio, and visual data into AI-friendly formats. Learn how to find the latest most popular models for tasks such as text generation and harness the power of pre-built pipelines.

Exercise 1: Hugging Face model navigation Exercise 2: How many models!?Exercise 3: Finding the most popular text-to-image model Exercise 4: Preprocessing different modalities Exercise 5: Text tokenizing Exercise 6: Image preprocessing Exercise 7: Audio preprocessing Exercise 8: Pipeline tasks and evaluations Exercise 9: Pipeline caption generation Exercise 10: Passing keyword arguments Exercise 11: Model evaluation on a custom dataset

Learn to master individual modalities with state-of-the-art models. Dive into computer vision for image classification and segmentation, explore speech recognition and text-to-speech synthesis, and learn effective fine-tuning techniques. Build practical skills with pre-trained models from Hugging Face's transformers library.

Exercise 1: Computer vision Exercise 2: Image classification Exercise 3: Object detection Exercise 4: Image background removal Exercise 5: Fine-tuning computer vision models Exercise 6: CV fine-tuning: dataset prep Exercise 7: CV fine-tuning: model classes Exercise 8: CV fine-tuning: trainer configuration Exercise 9: Speech recognition and audio generation Exercise 10: Automatic speech recognition Exercise 11: Creating speech embeddings Exercise 12: Audio denoising Exercise 13: Fine-tuning text-to-speech models Exercise 14: Fine-tuning a text-to-speech model Exercise 15: Generating new speech

Learn to fuse visual, textual, and audio information for richer AI applications. Master techniques like CLIP for zero-shot classification, build sentiment analyzers that see and read, and create emotion detectors that combine facial expressions with voice. Take your AI models beyond single-modality thinking.

Exercise 1: Bildklassifizierung ohne Trainingsdaten Exercise 2: Zero-Shot-Lernen mit CLIP Exercise 3: Automatisierte Qualitätsbewertung von Bildunterschriften

Aktuelle Übung

Exercise 4: Multimodale Stimmungsanalyse Exercise 5: Vision Language Models (VLMs) anregen Exercise 6: Multimodale Sentimentklassifizierung mit Qwen Exercise 7: Video-Klassifizierung ohne Trainingsdaten Exercise 8: Video-Audio-Trennung Exercise 9: Video-Stimmungsanalyse mit CLIP und CLAP

Transform ideas into reality! Master cutting-edge AI techniques to generate and manipulate visual content using text prompts. Create stunning images, edit photos intelligently, and build powerful question-answering systems for images and documents. Turn your creative vision into digital reality with multi-modal AI.

Exercise 1: Visual question-answering (VQA)Exercise 2: VQA with Vision Language Transformers (ViLTs)Exercise 3: Document VQA with LayoutLM Exercise 4: Image editing with diffusion models Exercise 5: Custom image editing Exercise 6: Image inpainting Exercise 7: Video generation Exercise 8: Build a video!Exercise 9: Assessing video generation performance Exercise 10: Congratulations!