TTT-Video: Technology for Long Video Generation
Researchers have recently released an open-source project called TTT-Video, a technology that breaks through the traditional time limitations of AI video generation, capable of producing coherent video content up to 63 seconds long. This technology solves content consistency issues in long video generation through the innovative Test-Time Training method.
Addressing Key Challenges in Video Generation
Currently, most AI video generation models can only create short video clips of 3-5 seconds. This is because the Transformer models used for video generation have quadratically increasing computational costs when processing long sequences due to their self-attention mechanism, making it inefficient to process long videos.
TTT-Video solves this problem in an innovative way: it retains the attention layers of the original pretrained model for local attention on each 3-second segment, while introducing special Test-Time Training layers to handle long-distance relationships in the global context.
Technical Implementation
The project is based on the CogVideoX 5B model (a diffusion Transformer for text-to-video generation) with key innovations including:
- Introducing TTT layers to process the global sequence and its reversed version, combining outputs through gated residual connections
- Extending context by interleaving each segment with text and video embeddings
- Training in stages: first fine-tuning at the original pretrained 3-second video length, then gradually training at video lengths of 9, 18, 30, and 63 seconds
TTT-Video model architecture: Processing global sequences through TTT layers combined with local attention mechanisms
The research team used the classic cartoon “Tom and Jerry” as a test case, generating stylistically consistent and coherently animated videos of about one minute in length, though limited by the 5B parameter size, there is still room for improvement in generation quality.
Impressive Generation Results
The most impressive aspect of TTT-Video is its ability to generate “Tom and Jerry” style animations up to one minute long in a single pass, with:
- No need for any editing, splicing, or post-processing
- Content that is completely original, with scenes that don’t exist in the original cartoon
- Coherent character actions, scene transitions, and storylines
Animation frames generated by TTT-Video in Tom and Jerry style
Significance for AI Creators
This technology means the following for AI creators using tools like ComfyUI:
- The potential for longer, more narrative AI video generation in the future
- Solutions to key issues of consistency and coherence in video generation
- The possibility for creators to create longer video content without manually splicing multiple segments