DreamX-Creator 1.0: Open 7B Joint Audio-Video Generation at 2K
AMap open-sources DreamX-Creator 1.0, a 7B joint audio-video generator with gated cross-modal attention and a 1-step 2K refiner, built on Wan2.2 and fully downloadable.
The DreamX-Creator 1.0 teaser showing joint audio-video generation results at 2K resolution.
A 7B generator that speaks video and audio at once
Given a first frame and a text prompt, the base generator jointly models modality-specialized video and audio streams. Two mechanisms drive the bidirectional interaction:
- Gated Cross-Modal Attention: cross-attention weights (
cross_attn_weights.safetensors, 2.56 GB) let each modality attend to the other, with a learned gate controlling how much information flows across. - Progressive Joint Training: the two streams are trained together so that lip movements, on-screen action and the soundtrack stay synchronized rather than being stitched after the fact.
On top of the base generator, an Audio-Video Reinforcement Learning stage with modality-aware multimodal feedback improves visual quality, audio quality, semantic consistency and fine-grained audio-video alignment.
One-step refinement to 2K
The second half of the system is an autoregressive 1-step 2K refiner (SR-DiT, 5B parameters). It super-resolves the generated video to 2K while preserving content, motion and audio-aligned timing. The refiner also works on external videos, so you can feed it any clip: audio passes through untouched.
The refiner ships with its own speed knobs: KV cache, window attention and latent upsampler variants (a FlashLatentUpsampler by default, plus an optional distilled LightVAE fast decoder) trade quality against latency.
What you can download
All weights are on Hugging Face and ModelScope, about 54 GB in total:
| Component | Contents |
|---|---|
creator/ | 7B joint generator: video DiT shards (~20 GB), audio DiT (5.7 GB), cross-attention weights (2.6 GB) |
audio_vae/ | CreatorDACVAE audio VAE (0.74 GB) |
refiner/ | SR-DiT 5B (10 GB), FlashLatentUpsampler, optional causal 2D upsampler and LightVAE decoder |
wan2.2_ti2v_5b/ | Shared Wan2.2-TI2V-5B dependencies: video VAE, UMT5-xxl text encoder and tokenizer |
The Wan2.2 directory can be reused directly from Wan-AI/Wan2.2-TI2V-5B, so only the three DreamX-specific folders are new downloads if you already run Wan2.2.
Inference is plain Python: each of the two stages has its own requirements.txt and a one-command script, with CPU offload and multi-GPU sequence-parallel options documented in the sub-READMEs.
Availability
There is no native ComfyUI node for DreamX-Creator yet: the weights are in diffusers layout and inference runs through the official repository. The roadmap lists distilled, fewer-step models as the next milestone.
Comments
Sign in with GitHub to join the discussion.