Wan-Dancer-14B: Minute-Scale Music-to-Dance Video Generation from Tongyi Lab

Overview

Tongyi Lab (Alibaba Group's AI research institute behind Qwen, Wan, and Tongyi Fun) has released Wan-Dancer-14B, an open-source model that generates high-quality, rhythmically synchronized dance videos from music and reference images. The model overcomes the conventional duration barrier of diffusion-based video generation, producing stable 720p video at 30fps for over one minute — a feat that existing models typically struggle with beyond 20 seconds.

![]

Key Capabilities

Wan-Dancer supports five dance genres with full musical synchronization:

The model accepts a reference image (showing the dancer's appearance and pose) and an audio file (music track), along with a text prompt describing the desired dance style. It then generates a full dance video matching the music's rhythm and structure.

Technical Approach

Wan-Dancer employs a hierarchical framework to achieve coherent long-form generation:

1. Global Keyframe Planning — The first stage generates 5-second keyframe segments that capture the full-track musical structure, ensuring global coherence across the entire duration.

2. Local Temporal Refinement — The second stage refines each segment into high-resolution 720p frames, adding fine details while maintaining temporal smoothness between segments.

Key technical innovations include:

• Time-Mapped RoPE Embeddings — Dynamic frame rate adaptation that precisely aligns motion timing with musical beats and structure.
• Optical-Flow-Based Loss Function — Enhances motion continuity and reduces jitter between adjacent frames.
• Motion-Speed Control — Preserves high-fidelity details during rapid movements and sharp transitions.
• Full-Track Musical Context — The global stage processes the entire music track for coherent long-range planning, rather than processing short windows.

The model architecture builds upon the Wan2.1 work and DiffSynth-Studio, utilizing 14 billion parameters to achieve high-quality outputs.

Model Availability

Wan-Dancer-14B is released under the Apache 2.0 license with model weights and inference code available on HuggingFace and ModelScope.

Hardware Requirements

The inference code was tested on 8 × NVIDIA A800 (80GB) GPUs. For local use, the model will require significant GPU memory given its 14B parameter size, though quantization and optimization techniques may enable reduced-footprint inference.

ComfyUI Integration

Wan-Dancer-14B is natively supported in ComfyUI. Get started quickly with the official Wan-Dancer workflow.

Make sure you have updated ComfyUI to the latest version (update guide). The required model weights are available from the Comfy-Org/Wan-Dancer repository on Hugging Face.

Required Model Files

Place the following files in your ComfyUI/models/ directory:

+--- diffusion_models/
|    +--- wan2.2_dancer_14b_global_fp8_scaled.safetensors
|    +--- wan2.2_dancer_14b_local_fp8_scaled.safetensors
+--- loras/
|    +--- lightx2v_I2V_14B_480p_cfg_step_distill_rank64_bf16.safetensors
+--- text_encoders/
|    +--- umt5_xxl_fp16.safetensors
+--- clip_vision/
|    +--- clip_vision_h.safetensors
+--- vae/
|    +--- Wan2_1_VAE_bf16.safetensors

Model files are hosted on the Comfy-Org/Wan-Dancer and Comfy-Org/Wan_2.1_ComfyUI_repackaged repositories.

Report Issues

• Runtime errors: ComfyUI/issues
• UI/frontend issues: ComfyUI_frontend/issues
• Workflow issues: workflow_templates/issues