Wan2.2 Dancer: Minute-Scale Music-to-Dance Video Generation by Tongyi Lab

Overview

Wan-Dancer-14B is an open-source model from Tongyi Lab (Alibaba Group's AI research institute behind Qwen, Wan, and Tongyi Fun) that generates high-quality, rhythmically synchronized dance videos from music and reference images. The model overcomes the conventional duration barrier of diffusion-based video generation, producing stable 720p video at 30fps for over one minute — a feat that existing models typically struggle with beyond 20 seconds.

The model accepts a reference image (showing the dancer's appearance and pose) and an audio file (music track), along with a text prompt describing the desired dance style. It then generates a full dance video matching the music's rhythm and structure.

Architecture

Wan-Dancer employs a hierarchical framework that decouples the generation process into two stages:

Stage 1: Global Keyframe Planning

The first stage generates 5-second keyframe segments that capture the full-track musical structure, ensuring global coherence across the entire duration. It processes the complete music track as context for long-range planning.

Stage 2: Local Temporal Refinement

The second stage refines each segment into high-resolution 720p frames, adding fine details while maintaining temporal smoothness between segments.

Key Technical Innovations

• Time-Mapped RoPE Embeddings: Dynamic frame rate adaptation that precisely aligns motion timing with musical beats and structure, enabling accurate synchronization between audio and video.
• Optical-Flow-Based Loss Function: Enhances motion continuity by penalizing jitter and abrupt transitions between adjacent frames, resulting in smoother dance movements.
• Motion-Speed Control: Preserves high-fidelity details during rapid movements and sharp transitions, preventing motion blur and artifacts common in long-form video generation.
• Full-Track Musical Context: Unlike models that process short audio windows, the global stage uses the entire music track for coherent long-range planning across the full dance duration.

The framework builds upon the Wan2.1 work, DiffSynth-Studio, and the Wan2.2 MoE architecture.

Dance Genres

Wan-Dancer supports five dance genres with full musical synchronization:

ComfyUI Integration

Wan-Dancer-14B is natively supported in ComfyUI 0.22.0+. Get started with the official Wan-Dancer workflow. Make sure you have updated ComfyUI to the latest version.

Required Model Files

Place the following files in your ComfyUI/models/ directory:

+--- diffusion_models/
|    +--- wan2.2_dancer_14b_global_fp8_scaled.safetensors
|    +--- wan2.2_dancer_14b_local_fp8_scaled.safetensors
+--- loras/
|    +--- lightx2v_I2V_14B_480p_cfg_step_distill_rank64_bf16.safetensors
+--- text_encoders/
|    +--- umt5_xxl_fp16.safetensors
+--- clip_vision/
|    +--- clip_vision_h.safetensors
+--- vae/
|    +--- Wan2_1_VAE_bf16.safetensors

Model files are hosted on the Comfy-Org/Wan-Dancer and Comfy-Org/Wan_2.1_ComfyUI_repackaged repositories.

Resources