Valutazione automatica della qualità delle didascalie

Hai classificato correttamente l'immagine dell'abito, ma quanto era buona la descrizione originale?

Maa Fab wrap with a Trendy design dress with Vibrant color for an elegant touch of Fabric completely Soft and Comfortable wear with amazing design of Solid Boat ? Neck Flared Dress to make a perfect addition to your wardrobe collection.

Ora userai il modello CLIP per dare una misura quantitativa dell'accuratezza di questa descrizione usando il punteggio CLIP. La didascalia (description), l'immagine (image), la classe ToTensor e la funzione clip_score() di torchmetrics sono già state caricate.

Questo esercizio fa parte del corso

Modelli multi-modali con Hugging Face

Visualizza il corso

Istruzioni dell'esercizio

Converti l'immagine in un tensore PyTorch con intensità comprese tra 0 e 255.
Usa la funzione clip_score() per valutare la qualità della didascalia confrontando image_tensor e description con il modello openai/clip-vit-base-patch32.

Esercizio pratico interattivo

Prova a risolvere questo esercizio completando il codice di esempio.

# Convert the image to a PyTorch tensor
image_tensor = ____()(image)*____

# Use the clip_score function to assess the quality of the caption
score = ____(____, ____, "____")

print(f"CLIP score: {score}")

Modifica ed esegui il codice

Questo esercizio fa parte del corso

Modelli multi-modali con Hugging Face

IntermediárioNível de habilidade

4.8+

Inizia il corso gratis

Navigate the Hugging Face model hub, transform raw text, audio, and visual data into AI-friendly formats. Learn how to find the latest most popular models for tasks such as text generation and harness the power of pre-built pipelines.

Exercise 1: Hugging Face model navigation Exercise 2: How many models!?Exercise 3: Finding the most popular text-to-image model Exercise 4: Preprocessing different modalities Exercise 5: Text tokenizing Exercise 6: Image preprocessing Exercise 7: Audio preprocessing Exercise 8: Pipeline tasks and evaluations Exercise 9: Pipeline caption generation Exercise 10: Passing keyword arguments Exercise 11: Model evaluation on a custom dataset

Learn to master individual modalities with state-of-the-art models. Dive into computer vision for image classification and segmentation, explore speech recognition and text-to-speech synthesis, and learn effective fine-tuning techniques. Build practical skills with pre-trained models from Hugging Face's transformers library.

Exercise 1: Computer vision Exercise 2: Image classification Exercise 3: Object detection Exercise 4: Image background removal Exercise 5: Fine-tuning computer vision models Exercise 6: CV fine-tuning: dataset prep Exercise 7: CV fine-tuning: model classes Exercise 8: CV fine-tuning: trainer configuration Exercise 9: Speech recognition and audio generation Exercise 10: Automatic speech recognition Exercise 11: Creating speech embeddings Exercise 12: Audio denoising Exercise 13: Fine-tuning text-to-speech models Exercise 14: Fine-tuning a text-to-speech model Exercise 15: Generating new speech

Learn to fuse visual, textual, and audio information for richer AI applications. Master techniques like CLIP for zero-shot classification, build sentiment analyzers that see and read, and create emotion detectors that combine facial expressions with voice. Take your AI models beyond single-modality thinking.

Exercise 1: Classificazione di immagini zero-shot Exercise 2: Zero-shot learning con CLIP Exercise 3: Valutazione automatica della qualità delle didascalie

Esercizio in corso

Exercise 4: Analisi del sentiment multimodale Exercise 5: Prompting dei modelli Vision Language (VLM)Exercise 6: Classificazione multimodale del sentiment con Qwen Exercise 7: Classificazione video zero-shot Exercise 8: Separare audio e video Exercise 9: Analisi del sentiment dei video con CLIP CLAP

Transform ideas into reality! Master cutting-edge AI techniques to generate and manipulate visual content using text prompts. Create stunning images, edit photos intelligently, and build powerful question-answering systems for images and documents. Turn your creative vision into digital reality with multi-modal AI.

Exercise 1: Visual question-answering (VQA)Exercise 2: VQA with Vision Language Transformers (ViLTs)Exercise 3: Document VQA with LayoutLM Exercise 4: Image editing with diffusion models Exercise 5: Custom image editing Exercise 6: Image inpainting Exercise 7: Video generation Exercise 8: Build a video!Exercise 9: Assessing video generation performance Exercise 10: Congratulations!