ViPE — Video Pose Engine for 3D Geometric Perception (Overview & Usage)

TL;DR ViPE is a practical, open-source spatial AI engine that annotates raw videos with camera intrinsics, camera trajectories, and dense, near‑metric depth maps. It supports pinhole, wide‑angle, and 360° panoramas; runs via a simple CLI; and includes a sizeable public dataset for research and benchmarking.

What is ViPE?

ViPE (Video Pose Engine for 3D Geometric Perception) is a video processing pipeline that turns in‑the‑wild videos into structured 3D signals:

• Camera intrinsics and motion (per‑frame calibrated poses)
• Dense, near‑metric depth maps over time
• Robust handling of diverse content: selfie videos, cinematic footage, dashcams
• Support for multiple camera models: pinhole, wide‑angle, 360° panoramas

This helps bridge a key bottleneck for spatial AI systems: getting consistent, precise 3D supervision from raw, unconstrained videos.

References:

• GitHub: https://github.com/nv-tlabs/vipe
• Whitepaper: https://research.nvidia.com/labs/toronto-ai/vipe/assets/paper.pdf
• Project page: https://research.nvidia.com/labs/toronto-ai/vipe

Why it matters for production workflows

• Pre‑viz & blocking: Recover camera trajectories and depth layers to plan moves, composites, and CG inserts.
• VFX and match‑move: Export to COLMAP‑compatible formats for downstream 3D tools and pipelines.
• AR occlusion & relighting: Use dense depth for object placement and light estimation.
• Dataset bootstrapping: Automatically annotate large video corpora to train or fine‑tune 3D/SLAM/NeRF‑style models.

Note: Output quality varies by footage quality (motion blur, rolling shutter, low light) and chosen pipeline options. Always validate with your target tasks.

Quick Start (CLI)

After installation, the CLI processes mp4 videos end‑to‑end:

# 1) Create environment and install
conda env create -f envs/base.yml
conda activate vipe
pip install -r envs/requirements.txt
pip install --no-build-isolation -e .

# 2) Run inference on a single video
vipe infer YOUR_VIDEO.mp4 \
  --output vipe_results \
  --visualize \
  --pipeline default

# Option: memory‑friendlier depth path (less temporal smoothing, more 3D consistency)
vipe infer YOUR_VIDEO.mp4 --pipeline no_vda

# 3) Inspect results (viser UI)
vipe visualize vipe_results/

For multi‑video batches and fine‑grained Hydra config control, use the provided run.py workflow:

# Full pipeline on a directory of videos
python run.py pipeline=default \
  streams=raw_mp4_stream \
  streams.base_path=YOUR_VIDEO_OR_DIR

# Pose‑only (skip depth alignment)
python run.py pipeline=default \
  streams=raw_mp4_stream \
  streams.base_path=YOUR_VIDEO_OR_DIR \
  pipeline.post.depth_align_model=null