JoyAI-Video-Edit: Real-Time Video Editing From JD Open Source

JD Open Source released JoyAI-Video-Edit, a real-time, instruction-guided video editing system for open-ended video streams. Given a live camera stream or uploaded video and a natural-language edit instruction, it edits frames causally as they arrive — without waiting for the full video, requiring a predefined length, or revisiting future frames. In the deployment benchmark, the full end-to-end pipeline reaches 30.19 FPS at 720×1280, pushing video editing from offline batch processing toward interactive streaming generation.

Case 1: edited output from the official example set

Real-Time Open-Ended Editing

The system combines an MLLM-based condition encoder, a causal video VAE, and a 16B-parameter multimodal diffusion transformer. It is trained and deployed as an autoregressive diffusion editor, then accelerated with aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling to sustain high-throughput 720p editing while reducing train-inference mismatch and accumulated temporal drift.

Case 2: edited output from the official example set

How It Works

JoyAI-Video-Edit processes the video as a causal stream: each chunk of frames is encoded and edited using only past context, so generation latency stays bounded regardless of video length. The autoregressive diffusion design lets the model maintain consistency across long streams, while bounded KV-state inference keeps memory and per-chunk compute stable during deployment. Distillation (aligned autoregressive distribution matching) and long-horizon optimization close the gap between training and inference-time behavior, reducing drift that typically accumulates over many edited frames.

Case 3: edited output from the official example set

Availability

JoyAI-Video-Edit does not have native ComfyUI support yet; it runs through the official Python deployment pipeline. The project repository provides the full setup:

1. Install dependencies with Python 3.10+ and pip install -r requirements.txt
2. Download the released weights from Hugging Face into deploy/deps/checkpoints/JoyAI-Video-Edit/ (the DIT checkpoint and VAE). MiMo-VL and ONNX detector files are external runtime dependencies — see DEPLOYMENT.md for details
3. Launch the server with cd deploy && bash run_server.sh, then open http://localhost:8080

For remote machines, bind the server to 0.0.0.0 and open the selected port, or use SSH port forwarding. Custom deployments can edit deploy/run_server.sh to set checkpoint paths, CUDA device placement, host, and port.

The technical report covers the architecture, distillation, and deployment techniques in detail.