How to Use Stable Diffusion ControlNet

ControlNet transforms Stable Diffusion from unpredictable art generation into a precise creative tool. It uses reference images like poses, edges, or depth maps to guide the AI, ensuring your generated images match specific compositions and structures.

1. Install ControlNet Extension. Open the Automatic1111 WebUI and navigate to Extensions > Available > Load from. Search for 'sd-webui-controlnet' and click Install. Restart the WebUI completely after installation to activate ControlNet functionality.
2. Download Required Models. Navigate to the ControlNet models folder in your Stable Diffusion directory (usually models/ControlNet). Download the specific ControlNet models you need: control_v11p_sd15_openpose for pose control, control_v11p_sd15_canny for edge detection, or control_v11f1p_sd15_depth for depth maps. Each model file is approximately 1.45GB.
3. Prepare Your Reference Image. Select or create a reference image that demonstrates the composition, pose, or structure you want. The image should be clear and well-defined. For pose control, use images with distinct human figures. For edge control, choose images with strong outlines and contrast.
4. Load ControlNet Interface. Scroll down in the WebUI to find the ControlNet section below the generation parameters. Click the arrow to expand ControlNet Unit 0. Check the 'Enable' checkbox to activate the ControlNet controls for this generation session.
5. Upload Reference Image and Select Preprocessor. Click the image upload area in ControlNet Unit 0 and select your reference image. Choose the appropriate preprocessor from the dropdown menu: 'openpose_full' for human poses, 'canny' for edge detection, or 'depth_midas' for depth maps. The preprocessor analyzes your reference image to extract control information.
6. Configure Model and Control Weight. Select the matching ControlNet model from the Model dropdown that corresponds to your chosen preprocessor. Set the Control Weight between 0.5 and 1.5, where 1.0 provides balanced control. Lower values give the AI more creative freedom, while higher values enforce stricter adherence to the reference.
7. Generate with ControlNet Guidance. Write your text prompt as usual in the main prompt field, then click Generate. ControlNet will guide the generation process using your reference image structure while incorporating your text prompt details. The AI creates new content that follows the reference composition but matches your written description.