Custom image editing

AI image generation is already pretty cool, but some models even support custom image editing, a multi-modal variant of image generation that takes both a text prompt and source image input. Have a go at modifying this famous self-portrait of Van Gogh to be of the cartoon character Snoopy using the StableDiffusionControlNetPipeline:

Famous Van Gogh painting

Note: Inference on diffusion models can take a long time, so we've pre-loaded the generated image for you. Running different prompts will not generated new images.

The Canny filter version of the image has been created for you (canny_image). The StableDiffusionControlNetPipeline and ControlNetModel classes have been imported from the diffusers library. The generator list (generator) has been created.

This exercise is part of the course

Multi-Modal Models with Hugging Face

Exercise instructions

Load the ControlNetModel from the lllyasviel/sd-controlnet-canny checkpoint.
Load the StableDiffusionControlNetPipeline from the runwayml/stable-diffusion-v1-5 checkpoint, passing the controlnet provided.
Run the pipeline using the prompt, canny_image, and the negative_prompt and generator provided.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

# Load a ControlNetModel from the pretrained checkpoint
controlnet = ____("____", torch_dtype=torch.float16)

# Load a pretrained StableDiffusionControlNetPipeline using the ControlNetModel
pipe = ____(
    "____", controlnet=____, torch_dtype=torch.float16
)
pipe = pipe.to("cuda")

prompt = ["Snoopy, best quality, extremely detailed"]

# Run the pipeline
output = pipe(
    ____,
    ____,
    negative_prompt=["monochrome, lowres, bad anatomy, worst quality, low quality"],
    generator=____,
    num_inference_steps=20,
)

plt.imshow(output.images[0])
plt.show()

Edit and Run Code

This exercise is part of the course

Multi-Modal Models with Hugging Face

IntermediateSkill Level

4.9+

Start Course for Free

Navigate the Hugging Face model hub, transform raw text, audio, and visual data into AI-friendly formats. Learn how to find the latest most popular models for tasks such as text generation and harness the power of pre-built pipelines.

Exercise 1: Hugging Face model navigation Exercise 2: How many models!?Exercise 3: Finding the most popular text-to-image model Exercise 4: Preprocessing different modalities Exercise 5: Text tokenizing Exercise 6: Image preprocessing Exercise 7: Audio preprocessing Exercise 8: Pipeline tasks and evaluations Exercise 9: Pipeline caption generation Exercise 10: Passing keyword arguments Exercise 11: Model evaluation on a custom dataset

Learn to master individual modalities with state-of-the-art models. Dive into computer vision for image classification and segmentation, explore speech recognition and text-to-speech synthesis, and learn effective fine-tuning techniques. Build practical skills with pre-trained models from Hugging Face's transformers library.

Exercise 1: Computer vision Exercise 2: Image classification Exercise 3: Object detection Exercise 4: Image background removal Exercise 5: Fine-tuning computer vision models Exercise 6: CV fine-tuning: dataset prep Exercise 7: CV fine-tuning: model classes Exercise 8: CV fine-tuning: trainer configuration Exercise 9: Speech recognition and audio generation Exercise 10: Automatic speech recognition Exercise 11: Creating speech embeddings Exercise 12: Audio denoising Exercise 13: Fine-tuning text-to-speech models Exercise 14: Fine-tuning a text-to-speech model Exercise 15: Generating new speech

Learn to fuse visual, textual, and audio information for richer AI applications. Master techniques like CLIP for zero-shot classification, build sentiment analyzers that see and read, and create emotion detectors that combine facial expressions with voice. Take your AI models beyond single-modality thinking.

Exercise 1: Zero-shot image classification Exercise 2: Zero-shot learning with CLIP Exercise 3: Automated caption quality assessment Exercise 4: Multi-modal sentiment analysis Exercise 5: Prompting Vision Language Models (VLMs)Exercise 6: Multi-modal sentiment classification with Qwen Exercise 7: Zero-shot video classification Exercise 8: Video audio splitting Exercise 9: Video sentiment analysis with CLIP CLAP

Transform ideas into reality! Master cutting-edge AI techniques to generate and manipulate visual content using text prompts. Create stunning images, edit photos intelligently, and build powerful question-answering systems for images and documents. Turn your creative vision into digital reality with multi-modal AI.

Exercise 1: Visual question-answering (VQA)Exercise 2: VQA with Vision Language Transformers (ViLTs)Exercise 3: Document VQA with LayoutLM Exercise 4: Image editing with diffusion models Exercise 5: Custom image editing

Current Exercise

Exercise 6: Image inpainting Exercise 7: Video generation Exercise 8: Build a video!Exercise 9: Assessing video generation performance Exercise 10: Congratulations!