Skip to content
Help Build a Better ComfyUI Knowledge Base Become a Patron
NewsAli Tongyi Lab Releases VACE: All-in-One Video Creation and Editing Model

Ali Tongyi Lab Releases VACE: All-in-One Video Creation and Editing Model

On March 11, Tongyi Lab released its new video creation and editing model VACE (Video Creation and Editing), an AI tool that integrates multiple video processing functions into one platform, aiming to simplify video creation through a unified framework.

Main Features

The core advantage of the VACE model lies in its “one-stop” processing capability, integrating complex tasks that traditionally require multiple tools working together into a single framework. Specific functions include:

Unified Multi-task Framework

  • Text-to-Video (T2V): Generate corresponding video content directly through text descriptions
  • Reference-to-Video (R2V): Generate videos containing specific subjects based on image or video samples
  • Video-to-Video (V2V): Implement video style conversion, dynamic element addition, and other global adjustments
  • Masked Video-to-Video (MV2V): Modify specific areas of video using spatiotemporal masks

Flexible Creation Combination Capabilities

The most distinctive feature of VACE is its support for “universal editing,” allowing users to flexibly combine different functions:

  • Move Anything: Adjust the motion trajectory of objects in the video
  • Swap Anything: Replace characters or objects in the video with specified references
  • Expand Anything: Extend video boundaries or fill in content
  • Animate Anything: Give natural movement effects to static images

Technical Highlights

The VACE model employs several innovative technologies:

  • Video Condition Unit: Unified processing of multimodal inputs such as text, images, video, and masks
  • Concept Decoupling Strategy: Automatically separate elements in video (such as characters, background, actions), supporting independent modification
  • Context Adapter Structure: Based on the Diffusion Transformer architecture, dynamically adjusting generation strategy to adapt to different tasks

Practical Application Scenarios

This model can be widely applied to:

  • Rapid production of social media short videos
  • Advertising and marketing content creation
  • Film post-production and special effects processing
  • Educational training video generation

Development Team

VACE was developed by the research team at Tongyi Lab, with core members including: Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu.

Future Development

The development team states that VACE will continue to be optimized in the future:

  • Improve video generation quality and coherence
  • Expand real-time editing capabilities
  • Enhance 3D generation features
  • Explore voice command interaction

The launch of VACE represents an important step in the development of AI video creation tools toward user-friendliness and integration, expected to significantly lower the threshold for video creation and provide content creators with more convenient tools.