Inferring a low-dimensional interpretable representation for macro motion in a dynamic scene.

Abstract

We study macro motion analysis, where macro motion refers to the collection of all visually observable motions in a dynamic scene. Traditional filtering-based methods on motion analysis typically focus only on local and tiny motions, yet fail to represent large motions or 3D scenes. Recent dynamic neural representations can faithfully represent motions using correspondences, but they cannot be directly used for motion analysis. In this work, we propose Phase-based neural polynomial Gabor fields (Phase-PGF), which learns to represent scene dynamics with low-dimensional time-varying phases. We theoretically show that Phase-PGF has several properties suitable for macro motion analysis. In our experiments, we collect diverse 2D and 3D dynamic scenes and show that Phase-PGF enables dynamic scene analysis and editing tasks including motion loop detection, motion factorization, motion smoothing, and motion magnification.

Method

A phase generator yields time-varying phases to modulate the Gabor basis in Phase-PGF, which produces a latent feature field for volume rendering. Please refer to the text in Sec. 3.3 of the paper for more descriptions.

Motion Analysis

Jelly

Bouncing

Collision

3 Balls

Projectile

Damping

Qualitative Comparison

Comparison to Dense Tracking Baseline

Motion Magnification

Motion Manipulation

Failure Cases

Citation

The website template was borrowed from Michaël Gharbi and Jon Barron.