ClipProj: Swap MiniMax H3's 32B Text Encoder for a 4B in ComfyUI

ComfyUI-ClipProj is a new custom node that replaces MiniMax H3's Qwen3-VL-32B text encoder with a much smaller Qwen3-VL-4B plus a learned linear projection, cutting the conditioning VRAM footprint from 15.7 GB to 4.5 GB (projection matrices on Hugging Face). The DiT, the VAEs and the sampler are untouched: the node returns an object that behaves like the official CLIP, so it drops into the existing clip input with no rewiring.

The complete pipeline: encoder, projection, conditioning, sampling, decode and video output (examples/minimax_h3_clipproj.json).

Why it saves so much VRAM

MiniMax H3 uses a Qwen3-VL-32B truncated to 50 layers — 15.7 GB in NVFP4 — solely to turn a prompt into a [seq, 5120] conditioning tensor. ClipProj replaces it with a Qwen3-VL-4B (2560 dims) plus a learned linear map into the 5120-dim space the DiT expects:

cond = ((h - mean_in) / std_in) @ W * std_out + mean_out

Measured encoder memory:

How the projection is learned

The 4B and 32B encoders share the same tokenizer (151,936 tokens), so a prompt yields identical tokens at identical positions in both. That makes a position-by-position mapping learnable. The calibration is ridge regression, not training: encode N prompts with both models, accumulate XᵀX and XᵀY in streaming (constant memory), then solve. No gradients, no epochs, no learning rate.

The author's measurements (MEASUREMENTS.md) report a cross-prompt cosine of ~0.71 with a 2,000-prompt corpus. In actual generation the projection holds up for simple prompts, structured multi-shot prompts (subject_definitions, timecoded shots, overall_soundscape), and fl2va with first and last frame. It is also robust to swapping encoder weights: a projection calibrated on bf16 works on an abliterated fp8 variant and on int8_convrot.

ComfyUI usage

Install the node:

cd ComfyUI/custom_nodes
git clone https://github.com/nicolab28/ComfyUI-ClipProj

Restart ComfyUI. There is no requirements.txt — the nodes import only torch and ComfyUI's own modules. On first launch the folder ComfyUI/models/clip_projections/ is created; put the projection matrices (mmh3-4b-ClipProj*.safetensors, mmh3-8b-ClipProj*.safetensors) there. Example workflows are in examples/: drag minimax_h3_clipproj.json onto the canvas.

An example workflow pairing the small encoder with structured H3 prompt rewriting (examples/rewrite_h3_prompt.json).

Status

Version 0.1.4 of the node frees the GPU card before loading a replacement encoder (fixing OOM spikes when two encoders would otherwise be resident at once), frees projection caches on reload, and keeps residual networks in their saved precision. The -mlp matrices are now fp16 and half the size (240 MB for the 4B, 288 MB for the 8B).

The project is explicitly a proof of concept: built and tested on a single setup (Windows 11, NVIDIA RTX 3090 / 4070 / 3060, ComfyUI 0.31.0) with deliberately limited exploration. Expect rough edges and breaking changes.

Availability

• Custom node: nicolab28/ComfyUI-ClipProj
• Projection matrices: NicoLab28/ClipProj-MiniMax-H3 on Hugging Face (4B and 8B variants, plus celeb and -mlp options)