Skip to content

Kuaishou and PKU Jointly Release Pyramidal Flow Matching Video Generation Model

Recently, the research teams from Kuaishou Technology and Peking University jointly introduced a new video generation model - Pyramidal Flow Matching. This model, based on flow matching autoregressive video generation technology, can produce high-quality, long-duration video content, marking a significant breakthrough in the field of video generation.

Overview of the Pyramidal Flow Matching Model

The Pyramidal Flow Matching model is a training-efficient autoregressive video generation model developed based on flow matching technology. Here are the main features of this model:

  1. Open-source Training Data: The model was trained exclusively using open-source datasets, utilizing a total of 20.7k hours of A100 GPU computing resources.
  2. High-Resolution Output: Capable of generating videos with a resolution of 1280x768.
  3. Long-Duration Generation: Supports the generation of videos up to 10 seconds long at 24 frames per second.
  4. Model Scale: Total parameter count of 2B (2 billion).

Model Capability Showcase

The Pyramidal Flow Matching model demonstrates various video generation capabilities, including text-to-video generation and image-based video generation. Here are some typical examples:

1. Text-to-Video Generation (1280x768, 10 seconds, 24FPS)

The model can generate realistic video scenes based on detailed text descriptions. For example:

  • Description: “Beautiful, snowy Tokyo city is bustling. The camera moves through the bustling city street, following several people enjoying the beautiful snowy weather and shopping at nearby stalls.”

    Tokyo Snow Scene Video

  • Description: “At dusk, a car is driving on the highway, with the rearview mirror reflecting a colorful sunset and serene scenery.”

    Highway Sunset Video

2. Text-to-Video Generation (1280x768, 5 seconds, 24FPS)

The model can also generate shorter but content-rich video clips:

  • Description: “A cat waking up its sleeping owner, demanding breakfast.”

    Cat Waking Owner Video

  • Description: “A drone camera circles around a beautiful historic church built on a rocky outcropping along the Amalfi Coast, the view showcases historic and magnificent architectural details and tiered pathways and patios.”

    Amalfi Coast Church Video

3. Image-Based Video Generation (1280x768, 5 seconds, 24FPS)

The model also has the ability to transform static images into dynamic videos:

  • Description: “A car driving on the road.”

    Car Driving Video

  • Description: “FPV flying over the Great Wall.”

    Great Wall Flying Video

Technical Highlights

  1. Flow Matching Technology: Adopts flow matching as the core technology, enhancing the coherence and realism of video generation.
  2. Pyramidal Structure: Uses a pyramidal structure to process spatiotemporal information in videos, effectively improving generation quality.
  3. Efficient Training: Achieves high-quality video generation using only open-source datasets with limited computational resources.
  4. Diverse Output: Supports video generation in various resolutions and durations, adapting to different application scenarios.

Potential Applications

The emergence of the Pyramidal Flow Matching model brings new possibilities to multiple fields:

  1. Creative Content Production: Provides new tools for creating advertisements, movie trailers, and other creative content.
  2. Education and Training: Rapidly generates educational videos or simulated scenarios.
  3. Game Development: Assists in creating game scenes and animations.
  4. Virtual Reality: Generates rich visual content for VR/AR applications.

Conclusion

The Pyramidal Flow Matching model, jointly developed by Kuaishou Technology and Peking University, represents the latest advancement in video generation technology. By combining flow matching and pyramidal structure, this model can generate high-quality, long-duration video content, bringing new possibilities to the field of AI video generation. As the technology further develops and finds applications, we can expect to see more stunning AI-generated video content.

Interested readers can visit the project’s official website to learn more details and personally experience the powerful capabilities of the model.