How to Use ControlNet for Image Generation in ComfyUI? 🚧

ControlNet is a powerful image generation control technology that allows users to precisely guide the AI model’s image generation process by inputting a conditional image. By using ControlNet, users can better control the AI image generation process and create images that better meet specific needs and imaginations.

Since ComfyUI does not have a built-in ControlNet model, you need to install the corresponding ControlNet model files before starting this tutorial. Like other types of models such as embedding, LoRA, etc., ControlNet has a version correspondence with the Checkpoint model, such as:

• If you are using the SD1.5 version of the checkpoint model, then the corresponding ControlNet should also support SD1.5.
• If you are using the Flux.1 model, then the corresponding ControlNet should also support Flux.1.

This tutorial will cover the following parts:

• A brief explanation of the functions and roles of the ControlNet model.
• How to install the controlNet model in ComfyUI (including corresponding model download channels).
• The use of different types of ControlNet models in ComfyUI.

Due to the many versions of ControlNet models, this tutorial only provides a general explanation of the installation method.

The Function and Role of ControlNet

Imagine that ControlNet is like an art director giving guidance to a painter during the painting process.

1. Canvas and brushes (base model): The Stable Diffusion model is like the painter’s canvas and brushes, capable of creating various images.

2. Art director (ControlNet): ControlNet is like an art director standing next to the painter, holding a reference image or sketch.

3. Guidance process:

   • The art director will tell the painter what to paint where on the canvas based on the reference image.
   • The painter (base model) still uses their own skills and style to paint but follows the suggestions of the art director.

4. Different types of guidance:

   • Outline guidance (such as Canny ControlNet): Like the art director holding a line drawing, guiding the painter on how to outline.
   • Pose guidance (such as Openpose ControlNet): It’s like the art director demonstrating the pose of the figure, allowing the painter to create accordingly.
   • Depth guidance (such as Depth ControlNet): As if the art director provides information on the three-dimensional sense of the scene, guiding the painter on how to represent depth.

5. Adjusting the influence:

   • Sometimes the painter will follow the art director’s suggestions more closely (high control intensity).
   • Sometimes the painter will exercise more of their own creativity (low control intensity).

6. Multiple guidance:

   • It can be imagined that there are multiple art directors giving advice on different aspects, and the painter needs to balance these opinions.

In this way, ControlNet allows us to more accurately control the process of AI-generated images, just as an art director helps the painter create works that better meet specific requirements.

Download and Installation of ControlNet Model

After understanding the basic concepts, we need to install the corresponding ControlNet model files first. Here, I have compiled some ControlNet download resources for you to choose the controlNet that matches the version of Checkpoint you are currently using.

Downloading the ControlNet Model

You can find the corresponding ControlNet model in the resource section.

How to Install the ControlNet Model?

1. Install ControlNet for ComfyUI Only

After downloading the model, you need to place the files in the /ComfyUI/models/controlnet folder.

2. Share ControlNet Model with WebUI

If you are sharing model files with WebUI, please ensure that the corresponding file configurations are correct.

1. Locate the configuration file for additional model files in ComfyUI: /ComfyUI/extra_model_paths.yaml
2. Open extra_model_paths.yaml with a text editor.
3. Check the configuration file information as follows:

#all you have to do is change the base_path to where yours is installed
a111:
    base_path: D:\sd-webui-aki-v4.2 # Your WebUI directory
 
    checkpoints: models/Stable-diffusion
    configs: models/Stable-diffusion
    vae: models/VAE
    loras: |
         models/Lora
         models/LyCORIS
    upscale_models: |
                  models/ESRGAN
                  models/RealESRGAN
                  models/SwinIR
    embeddings: embeddings
    hypernetworks: models/hypernetworks
    controlnet: models/ControlNet # The model should be installed in this folder
    ipadapter: models/ipadapter
 

You should pay attention to the configuration options for controlnet. For example, in my configuration file, the path for my ControlNet installed model should be D:\sd-webui-aki-v4.2\models\ControlNet. Only by matching the configuration can you ensure that ComfyUI can find the corresponding model files.

How to use ControlNet in ComfyUI?

🚧