Using the generated temporal object intrinsics, we can insert them to any 3D scene, render them from any viewpoint, under any environmental lighting conditions, at any time of their lifespan.
Using the generated temporal object intrinsics, we can insert them to any 3D scene, render them from any viewpoint, under any environmental lighting conditions, at any time of their lifespan.
Starting from a bud, the roses gradually expands their pedals and blossoms into full glory, only to wither and conclude their lifespans.
We show the generated temporal object intrinsics using the proposed pipeline in this section.
We show the comparison of the generated 4D contents and baselines. Note that none of the baselines could generate object intrinsics or relightable graphics assets. Therefore we only compare the rendering.
View 1
View 2
For a modeled natural process, we first estimate a task-specific Neural Template to get a 4D-consistent representation of its temporal state changes. Then we iteratively optimize a hybrid 4D intrinsics field to generate temporal object intrinsics. In each iteration of optimization, we sample a camera pose and timestamp to render this optimizable 4D field into intrinsic maps. These maps are subsequently rendered into RGB images using a physically-based renderer. Concurrently, the same camera pose and timestamp are used to query the Neural Template for the corresponding Neural State Map. This map conditions a fine-tuned 2D diffusion model, which provides guidance signals to update the 4D representation. For more details, please refer to our paper.
@article{geng2024rose4d,
title={Birth and Death of a Rose},
author={Chen Geng and Yunzhi Zhang and Shangzhe Wu and Jiajun Wu},
journal={arXiv preprint arXiv:2412.05278},
year={2024}
}