1. Learn
    2. /
  2. Courses
    3. /
  3. Multi-Modal Models with Hugging Face
Connected

Exercise

Custom image editing

AI image generation is already pretty cool, but some models even support custom image editing, a multi-modal variant of image generation that takes both a text prompt and source image input. Have a go at modifying this famous self-portrait of Van Gogh to be of the cartoon character Snoopy using the StableDiffusionControlNetPipeline:

Famous Van Gogh painting

Note: Inference on diffusion models can take a long time, so we've pre-loaded the generated image for you. Running different prompts will not generated new images.

The Canny filter version of the image has been created for you (canny_image). The StableDiffusionControlNetPipeline and ControlNetModel classes have been imported from the diffusers library. The generator list (generator) has been created.

Instructions

100 XP
  • Load the ControlNetModel from the lllyasviel/sd-controlnet-canny checkpoint.
  • Load the StableDiffusionControlNetPipeline from the runwayml/stable-diffusion-v1-5 checkpoint, passing the controlnet provided.
  • Run the pipeline using the prompt, canny_image, and the negative_prompt and generator provided.