Tracking Everything Everywhere All at Once

We present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estim...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Wang, Qianqian, Yen-Yu, Chang, Cai, Ruojin, Li, Zhengqi, Hariharan, Bharath, Holynski, Aleksander, Snavely, Noah
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 12.09.2023
Subjects
Algorithms
Consistency
Estimation
Motion simulation
Object motion
Optical flow (image analysis)
Optimization
Pixels
Representations
Tracking
Trajectory analysis
Online AccessGet full text

Cover

More Information
Summary:We present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively. See our project page for more results: http://omnimotion.github.io/
ISSN:2331-8422