FloED: Open Source Efficient Video Inpainting with Optical Flow-Guided Diffusion
Recently, a research team from Hong Kong University of Science and Technology and Alibaba DAMO Academy released FloED, a video inpainting framework that uses optical flow-guided diffusion model technology to bring higher temporal coherence and computational efficiency to video inpainting tasks. The team has also open-sourced the inference code and model weights, providing a new tool for the AI video processing field.
Core Problems Addressed
Video inpainting refers to using AI technology to fill in missing or removed parts of videos, making the results look natural and coherent. Existing diffusion model-based video inpainting methods have two main problems:
- Insufficient temporal coherence: Repaired videos often exhibit flickering and instability between frames
- Low computational efficiency: Diffusion models are computationally expensive, and the additional steps required for video processing make existing methods slow
FloED solves these problems through clever design, making video inpainting results more natural while significantly improving processing efficiency.
Technical Innovations
FloED’s main innovations include:
-
Dual-branch architecture: A dedicated branch first restores damaged optical flow (motion information of objects in the video), then provides this motion information to the main inpainting branch through multi-scale flow adapters, guiding the direction of generated content
-
Latent space interpolation acceleration: Uses optical flow information for feature interpolation, accelerating the multi-step denoising process without additional training
-
Flow attention caching: Optimizes the computation process, reducing the additional computational overhead introduced by optical flow
These technologies allow FloED to maintain high-quality video inpainting results while significantly increasing processing speed.
Application Scenarios and Demonstrations
FloED is mainly applicable to two types of video inpainting tasks:
Object Removal
Removing unwanted objects from videos while maintaining background coherence and natural transitions.
Background Restoration
Repairing large areas of background in videos, maintaining visual and temporal coherence with the surrounding environment.
Open Source Progress
According to the latest project updates, the FloED team released inference code and model weights on April 13, 2025. Interested users can access them through the following steps:
- Install the required environment (via environment configuration file)
- Download and prepare FloED weights
- Use the provided example scripts to quickly start inference
Future Plans
According to the project team’s roadmap, the following will be released in the future:
- Latent space interpolation code
- Training code and evaluation benchmarks