Ali Tongyi Lab Releases VACE: All-in-One Video Creation and Editing Model
On March 11, Tongyi Lab released its new video creation and editing model VACE (Video Creation and Editing), an AI tool that integrates multiple video processing functions into one platform, aiming to simplify video creation through a unified framework.
Main Features
The core advantage of the VACE model lies in its “one-stop” processing capability, integrating complex tasks that traditionally require multiple tools working together into a single framework. Specific functions include:
Unified Multi-task Framework
- Text-to-Video (T2V): Generate corresponding video content directly through text descriptions
- Reference-to-Video (R2V): Generate videos containing specific subjects based on image or video samples
- Video-to-Video (V2V): Implement video style conversion, dynamic element addition, and other global adjustments
- Masked Video-to-Video (MV2V): Modify specific areas of video using spatiotemporal masks
Flexible Creation Combination Capabilities
The most distinctive feature of VACE is its support for “universal editing,” allowing users to flexibly combine different functions:
- Move Anything: Adjust the motion trajectory of objects in the video
- Swap Anything: Replace characters or objects in the video with specified references
- Expand Anything: Extend video boundaries or fill in content
- Animate Anything: Give natural movement effects to static images
Technical Highlights
The VACE model employs several innovative technologies:
- Video Condition Unit: Unified processing of multimodal inputs such as text, images, video, and masks
- Concept Decoupling Strategy: Automatically separate elements in video (such as characters, background, actions), supporting independent modification
- Context Adapter Structure: Based on the Diffusion Transformer architecture, dynamically adjusting generation strategy to adapt to different tasks
Practical Application Scenarios
This model can be widely applied to:
- Rapid production of social media short videos
- Advertising and marketing content creation
- Film post-production and special effects processing
- Educational training video generation
Development Team
VACE was developed by the research team at Tongyi Lab, with core members including: Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu.
Future Development
The development team states that VACE will continue to be optimized in the future:
- Improve video generation quality and coherence
- Expand real-time editing capabilities
- Enhance 3D generation features
- Explore voice command interaction
The launch of VACE represents an important step in the development of AI video creation tools toward user-friendliness and integration, expected to significantly lower the threshold for video creation and provide content creators with more convenient tools.