Edits a playing video in real time from a typed instruction

You load a video, or point a camera, and type an instruction: change the background to a beach, make the jacket red, restyle everything as watercolor. The model rewrites frames as they arrive, so the edited video streams back to you in real time instead of rendering offline and appearing minutes later.

That streaming design is what is new here. Most editing models need the whole clip up front and revisit every frame; this one works frame by frame, which is why it can also run on footage with no fixed end, like a live feed. It covers subject and background changes, local edits, style transfer, motion changes, and reference-guided edits where you add an image of the subject or style you want carried over.

Under the hood it is a 16 billion parameter video model from JD's open-source team, released with full deployment code under the Apache 2.0 license. On server hardware the complete workflow runs at 30 frames per second at 720p.

Start with the live demo on Hugging Face: upload a short clip, type a background swap, and watch how quickly the edit appears. Then run a style instruction on the same clip and check whether the subject stays consistent; drift over longer clips is the known weak spot, so that second test tells you the most. If you have a high-end consumer card, the deployment guide gets the same streaming setup running locally; a single RTX 5090 manages real time at a reduced resolution.

Real time comes with tradeoffs: on consumer hardware the resolution drops well below 720p, and on long streams the picture can slowly drift away from the source, which is why the newest checkpoint focuses on identity preservation. The demo runs on a single GPU, so expect a queue when it is busy.

• Swap the background of a talking-head clip without a green screen.
• Restyle gameplay or drone footage as it plays to preview a look.
• Prototype a visual effect by typing it instead of building it in an editor.
• Carry a reference image's style or character onto your own footage.
• Build live camera filters that follow a written instruction.

Try the demo

View model page

Read this issue on neuralexpedition.com →