Omni-Effects - Unified and Spatially-Controllable Visual Effects Generation (Overview)
Download printable cheat-sheet (CC-BY 4.0)12 Aug 2025, 00:00 Z
TL;DR Omni-Effects unifies promptable and spatially controllable visual effects inside one CogVideoX-based pipeline. LoRA-MoE experts keep per-effect quality high, Spatial-Aware Prompts with Independent-Information Flow isolate each mask, and the Omni-VFX dataset plus released checkpoints make composite VFX runs practical for in-house teams.
What is Omni-Effects?
Omni-Effects is a research framework for controllable VFX generation that was unveiled on 11–12 August 2025 via arXiv, GitHub, and a LinkedIn announcement. Instead of training one LoRA per effect, the team introduces a unified diffusion pipeline that produces multiple effects at once while holding spatial constraints.
The system fine-tunes CogVideoX (image-to-video) backbones with two core ideas: a LoRA-based Mixture of Experts (LoRA-MoE) that routes prompts to effect-specific adapters, and a Spatial-Aware Prompt (SAP) format that injects mask layouts into the text stream. An Independent-Information Flow (IIF) block keeps those control signals from bleeding across effects. The release arrives with Omni-VFX, a curated VFX dataset distilled from Open-VFX assets, Remade-AI clips, and First–Last Frame-to-Video synthesis, plus CogVideoX checkpoints finetuned on that corpus.
Links:
- Project page: https://amap-ml.github.io/Omni-Effects.github.io/
- Hugging Face weights: https://huggingface.co/GD-ML/Omni-Effects
- Omni-VFX dataset: https://huggingface.co/datasets/GD-ML/Omni-VFX
Key ideas
- LoRA-MoE routing: groups expert LoRAs per effect category so the unified model can blend creative prompts without cross-task interference (paper + README).
- Spatial-Aware Prompting: appends binary masks to the text tokens, letting users paint regions for “Melt it”, “Levitate it”, “Explode it”, “Anime style”, or “Winter scene” in the same clip (project page + README).
- Independent-Information Flow: isolates control signals for each mask, stopping leakage when multiple effects fire simultaneously (paper abstract).
- Omni-VFX corpus: assembles edited assets, Remade-AI distillations, and FLF2V-generated clips to supply diverse training data and benchmarking splits (README).
- Released finetunes: CogVideoX-1.5 (prompt-guided) and CogVideoX-5B (single + multi-VFX) checkpoints, plus LoRA weights for spatial control (README updates).
