ControlNet XL Union: six kinds of control image in one model, and where to run it

Apache 2.0, weights free, about 8GB of video memory, which puts it on a mainstream graphics card. Self-hosting means the Hugging Face diffusers library. Note that this is an adapter, not a generator in its own right: it steers Stable Diffusion XL and needs that model underneath. In your browser it runs on CNAPS Studio, included in the basic plan, with the base model already there.

The reason this one exists is that control adapters normally come one per input type. An edge adapter, a depth adapter, a pose adapter, each a separate file to load and swap. This model handles six kinds of control image with the same set of weights, at no extra size or memory cost over a single-purpose adapter, and it was trained to combine two of them at once.

The cost is a little sharpness. On edges specifically, the dedicated canny model in this catalog holds structure more tightly, because its whole capacity goes to one job. Two ControlNet models sit in the CNAPS Studio catalog, and this is the flexible one.

What it actually is

A ControlNet adapter for Stable Diffusion XL with two additions over the original design: parameters shared across conditioning types rather than duplicated per type, and a fusion step that lets several control images feed one generation without extra compute.

Six conditioning types are exposed in CNAPS Studio, and the control image has to match the type you pick. Canny takes an edge map from the Canny Edge Annotator. Depth takes a depth map from Depth Anything Annotator. Openpose takes a skeleton, which is exactly what RF-DETR Keypoint produces. Hed takes a soft-edge map, gentler than canny. Normal takes a surface direction map. Segment takes a semantic mask that says what goes where.

Training used more than 10 million filtered and captioned images with bucket training, which is what lets it output any aspect ratio rather than only squares. Settings are guidance strength from 0.0 to 2.0 defaulting to 0.8, with 0.7 to 1.0 the usual range, diffusion steps defaulting to 25, a seed, an output size of 512, 768 or 1024, and an optional negative prompt.

Model typeMulti-condition ControlNet adapter for Stable Diffusion XLConditioning types in CNAPS StudioCanny, depth, openpose, soft edge, normal, segmentationCombine two at onceYes, learned during trainingParameter overheadNone over a single-condition adapterTraining dataMore than 10 million filtered and captioned imagesGuidance strength0.0 to 2.0, default 0.8; 0.7 to 1.0 works for most inputsDiffusion stepsDefault 25; 20 to 30 gives good qualityOutput size512, 768 or 1024 pixels, any aspect ratioVideo memoryAbout 8GBLicenseApache 2.0

What is published, and what is not

There is no benchmark table for this model. No aesthetic score, no control-fidelity measurement, nothing to rank it against alternatives. Its sibling in this catalog does publish two such figures, which makes the absence here more visible than it would otherwise be.

What is published is architecture and scale: the shared-parameter design, the fusion mechanism, the training set of more than 10 million captioned images, bucket training for arbitrary aspect ratios, and the claim that supporting many conditioning types costs no extra parameters. That last point is checkable in the file size and is the main engineering argument.

The one comparative claim worth carrying is the author's own, and it cuts against this model: the dedicated canny variant adheres more sharply on edge tasks. A vendor pointing at their own other model as better for a specific job is more credible than most benchmark tables, and it is the sentence to remember when choosing between the two.

How it compares to the other control model here

ControlNet XL UnionControlNet XL CannyConditioning typesSix: canny, depth, pose, soft edge, normal, segmentationCanny edges onlyStack two controls at onceYesNoStructural adherence on edgesGood; capacity is shared across typesSharper, per the authorDefault guidance strength0.81.0Aspect ratiosAny, thanks to bucket training512, 768 or 1024Video memoryAbout 8GBAbout 8GBPublished scoresNone6.03 aesthetic, 0.4200 perceptual similarityLicenseApache 2.0Apache 2.0

Only one of these two published any score, so that row is not a comparison. Everything else in the table is a documented specification difference rather than a measured quality difference.

Choosing is about the control signal. Reach for ControlNet XL Union whenever the input is anything other than edges: a depth map to keep a room's 3D layout while restyling it, a pose skeleton to put a character in a specific stance, a segmentation mask to fix what goes where. Reach for it also when you want two signals together, such as pose plus depth. Reach for ControlNet XL Canny when edges are the whole point and you want them held as tightly as possible.

What to chain it with

It takes a control image plus a text prompt and returns an image up to 1024 pixels. Since you rarely have a control image lying around, the model in front of it is part of the setup.

Three pairings cover most work. Canny Edge Annotator into canny mode restyles a scene while keeping its outlines. Depth Anything Annotator into depth mode restyles a space while keeping its geometry, which is the right route for interiors and architecture. And RF-DETR Keypoint into openpose mode takes a photographed pose and generates an entirely new character holding it, which is character work without a rig.

Afterwards, PiSA-SR or SMFANet+ takes 1024 pixels up to print size, and QWEN-Inpaint repairs one region without touching the rest.

Open ControlNet XL Union in CNAPS Studio, run the same photo through canny mode and through depth mode with an identical prompt, and see which kind of control your material actually needs.

Sources

• huggingface.co/xinsir/controlnet-union-sdxl-1.0 for the Apache 2.0 license, the base model, the shared-parameter and fusion design, the conditioning types, the training scale, bucket training, and the author's note that the dedicated canny model adheres more sharply.
• github.com/xinsir6/ControlNetPlus and arXiv:2302.05543 for the implementation and the original ControlNet method.
• huggingface.co/xinsir/controlnet-canny-sdxl-1.0 for the dedicated variant's published scores.
• docs.cnaps.ai/ai-model-hub for the six conditioning types exposed in CNAPS Studio, the settings and their defaults, and the annotators that produce control images.