เริ่มต้นใช้งานเริ่มต้นใช้งานได้ฟรี

Assessing video generation performance

You can assess the performance of your video generation pipelines using a multi-modal CLIP model, which tests the similarity between each video frame image and the prompt. You will use this to assess just how well your generated video from the previous exercise matches the prompt.

The load_video() function has been imported from diffusers.utils for you. The clip_score module has also been imported from torchmetrics.

แบบฝึกหัดนี้เป็นส่วนหนึ่งของหลักสูตร

Multi-Modal Models with Hugging Face

ดูคอร์ส

คำแนะนำการฝึกหัด

  • Set up a CLIP scoring function called clip_score_fn() from the clip_score() metric.
  • Calculate the CLIP score between each frame tensor in frame_tensors and prompt.

แบบฝึกหัดเชิงโต้ตอบแบบลงมือทำ

ลองทำแบบฝึกหัดนี้โดยเติมโค้ดตัวอย่างนี้ให้สมบูรณ์

# Setup CLIP scoring
clip_score_fn = partial(____, model_name_or_path="openai/clip-vit-base-patch32")

frame_tensors = []
for frame in frames:
    frame = np.array(frame)
    frame_int = (frame * 255).astype("uint8")
    frame_tensor = torch.from_numpy(frame_int).permute(2, 0, 1)
    frame_tensors.append(frame_tensor)

# Pass a list of CHW tensors as expected by clip_score
scores = clip_score_fn(____, [____] * len(frame_tensors)).detach().cpu().numpy()

avg_clip_score = round(np.mean(scores), 4)
print(f"Average CLIP score: {avg_clip_score}")
แก้ไขและรันโค้ด