MiniMax H3 Fun ControlNet Union: Canny, Depth, Pose and More

What it is

The checkpoint holds only the control branch (control_proj_in plus five control_blocks, about 6.8 GB) and is loaded on top of the base MiniMax H3 transformer. Control signals are injected at five of the 50 transformer blocks (layers 0, 10, 20, 30 and 40), and every control skip joins the main branch through a zero-gated projection.

• Union control — one checkpoint handles Canny, Depth, HED, MLSD and Pose control videos for video-to-video generation.
• Guidance-distilled — run with guidance_scale = 1.0 and a single forward pass per step; no classifier-free guidance needed.
• Inpainting — the control input is widened to 49 channels (latent + masked latent + mask); use the dedicated predict_v2v_control_inpaint.py example.
• Control strength — control_context_scale scales every control skip before it is added to the main branch: 1.0 gives the strongest control, lower values weaken the guidance of the control video, 0.0 switches the control branch off.

Generation follows the control video: the frame count snaps down to the largest 17 × n + 5 the video VAE can decode (duration capped at 15 seconds), the canvas keeps the control video's own aspect ratio within the height × width pixel budget, at a fixed 24 fps. Detailed prompts describing the scene, subject and camera give the most stable results.

Results

All samples below were generated with 40 inference steps, guidance_scale = 1.0 and control_context_scale = 1.00. The first video is the control input, the second is the generated output.

Canny

Canny control input (Tokyo street)

Generated output following the Canny structure

Pose

Pose control input (dance)

Generated output following the dance pose

Availability

The checkpoint currently runs through the VideoX-Fun inference pipeline. Clone the VideoX-Fun repository, place the base MiniMax-H3 model and this checkpoint under models/Diffusion_Transformer/, then edit the settings at the top of examples/minimax_h3_fun/predict_v2v_control.py and run it:

model_name          = "models/Diffusion_Transformer/MiniMax-H3"
config_path         = "config/minimax_h3/minimax_h3_control.yaml"
transformer_path    = "models/Diffusion_Transformer/MiniMax-H3-Fun-Controlnet-Union/MiniMax-H3-Fun-Controlnet-Union.safetensors"
control_video       = "your_control_video.mp4"
prompt              = "your prompt"

python examples/minimax_h3_fun/predict_v2v_control.py

Important setup notes from the model card:

• The config must build the control branch exactly as trained (control_blocks_places: [0, 10, 20, 30, 40], control_in_dim: 49, control_apply_audio: false); a mismatched layout makes the checkpoint fail to load.
• The checkpoint is guidance-distilled: keep guidance_scale = 1.0; values above 1 apply guidance twice and degrade the output.
• The control checkpoint carries only the control branch — the base MiniMax-H3 weights must be present.
• The transformer (~62 GB) plus the Qwen3-VL text encoder (~62 GB) do not fit a single 80 GB GPU fully loaded; use model_group_offload or model_cpu_offload_and_qfloat8.

There is no native ComfyUI loader for H3 ControlNet yet — the release targets the VideoX-Fun pipeline. A Gradio showcase Space demonstrates the model with the five published result pairs.