ControlNet XL Canny: keeping a photo's outlines while changing everything else, and where to run it

Apache 2.0, weights free, and the hardware ask is modest for image generation: about 8GB of video memory, which a mainstream graphics card has. Self-hosting means the Hugging Face diffusers library. One thing to know before you start is that this is an adapter rather than a whole generator: it steers Stable Diffusion XL and needs that model underneath. In your browser it runs on CNAPS Studio, included in the basic plan, with the base model already in place.

What it does is separate a picture into two things you can control independently. You give it an edge map for the geometry and a text prompt for everything else, and it generates a new image that keeps the first and replaces the second. Same room, same camera angle, same product silhouette, rendered as watercolour or as a photograph or in a different era.

The cost is that you have to produce the edge map first, and that this model only understands edges. Two ControlNet models sit in the CNAPS Studio catalog, and this is the specialist: it holds outlines more tightly than the general-purpose one, and does nothing else.

What it actually is

A ControlNet adapter trained only for canny edge maps. A canny edge map is a black image with white lines where the source photo had boundaries, and it is what the Canny Edge Annotator in the catalog produces from an ordinary photograph.

Training used more than 10 million filtered and captioned images, on a large cluster with an effective batch size of 2,560, in a single multi-resolution stage. Being trained on one conditioning type is the whole argument for it: a model that only ever sees edges gets better at edges than one splitting its capacity across six input types.

Six settings decide the result, and one of them matters more than the rest. Guidance strength runs 0.0 to 2.0 with a default of 1.0, where 0 ignores the edges and 2.0 follows them rigidly; 0.8 to 1.2 is the working range. Diffusion steps run 20 to 30, defaulting to 25. Guidance scale defaults to 5, and 5.0 to 7.5 is typical. There is a seed for reproducibility, an output size of 512, 768 or 1024, and an optional negative prompt.

Model typeControlNet adapter for Stable Diffusion XL, canny onlyConditioning inputCanny edge map, white lines on blackTraining dataMore than 10 million filtered and captioned imagesGuidance strength0.0 to 2.0, default 1.0; 0.8 to 1.2 works for most tasksDiffusion steps20 to 30, default 25Guidance scaleDefault 5; 5.0 to 7.5 is typicalOutput size512, 768 or 1024 pixelsVideo memoryAbout 8GBLicenseApache 2.0

The published numbers

Unusually for this category, there are two, and both come from the author's own comparison against two other open canny ControlNets.

The first is an aesthetic score, a learned rating of how good a picture looks, where this model scores 6.03 against 5.93 and 5.82 for the two alternatives. The second measures control rather than beauty: a perceptual similarity figure comparing the generated image's structure to the control input, where lower is better and this model scores 0.4200 against 0.5053 and 0.5387. On that second measure the margin is wide, and it is the one that matters if you chose a canny model to hold structure.

Both are vendor-reported, produced by the person who trained this model, and the two comparison models are not in the CNAPS Studio catalog so you cannot check the claim here. What makes them more useful than most vendor claims is that they are measurements against named alternatives rather than adjectives, and that the aesthetic gap is small while the control gap is large, which is a more honest shape than a clean sweep would be.

How it compares to the other control model here

ControlNet XL CannyControlNet XL UnionConditioning typesCanny edges onlySix in CNAPS Studio: canny, depth, pose, soft edge, normal, segmentationStructural adherence on edgesSharper, it is a dedicated modelGood, but the capacity is shared across typesDefault guidance strength1.00.8Diffusion steps20 to 30, default 25Default 25Stack several controls at onceNoYes, trained for itVideo memoryAbout 8GBAbout 8GBPublished scores6.03 aesthetic, 0.4200 perceptual similarityNoneLicenseApache 2.0Apache 2.0

The two scores in the last row cannot be compared across the columns, because only one of these models published any. What is documented rather than measured is that the dedicated model holds edges more sharply, which is what you would expect and what the author states.

Choosing is simple. Reach for ControlNet XL Canny when your control signal is edges and structural fidelity is the point: line art, architectural drawings, product silhouettes, anything where a wandering outline ruins the result. Reach for ControlNet XL Union when you want depth, pose or segmentation instead, or when you want to combine two control signals at once.

What to chain it with

It takes an edge map and a text prompt, and returns an image up to 1024 pixels. The edge map is the input you have to manufacture, which makes the model in front of it part of the setup rather than an optional extra.

Two chains cover the ground. The standard one is Canny Edge Annotator into this model: a photograph becomes an edge map, the edge map plus a prompt becomes a restyled version of the same scene. That is the whole style-transfer-with-geometry-intact route in two steps. The other starts earlier, with a sketch you drew yourself, which needs no annotator at all since line art already is an edge map.

After generation, PiSA-SR or SMFANet+ takes the 1024-pixel result up to print size, and QWEN-Inpaint fixes any single region that came out wrong without regenerating the whole picture.

Open ControlNet XL Canny in CNAPS Studio, run one photo at guidance strength 0.6 and again at 1.2, and watch how much freedom the prompt gains as the edges loosen.

Sources

• huggingface.co/xinsir/controlnet-canny-sdxl-1.0 for the Apache 2.0 license, the base model, the training scale and compute, the aesthetic score of 6.03, the perceptual similarity of 0.4200, and the comparison against two other open canny models.
• huggingface.co/xinsir/controlnet-union-sdxl-1.0 for the Union variant's conditioning types and defaults.
• docs.cnaps.ai/ai-model-hub for the CNAPS Studio settings, their ranges and defaults, and the annotators that produce control images.