Multi-view Performance Capture of Surface Details

• Published in: International journal of computer vision Vol. 124; no. 1; pp. 96 - 113
• Main Authors: Robertini, Nadia, Casas, Dan, De Aguiar, Edilson, Theobalt, Christian
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.08.2017; Springer; Springer Nature B.V
• Subjects: Artificial Intelligence; Computer Imaging; Computer Science; Consistency; Deformation effects; Derivatives; Digital video; Displacement; Error analysis; Exact solutions; Finite element method; Formability; Global optimization; Human performance; Image Processing and Computer Vision; Image processing systems; Mathematical analysis; Occlusion; Pattern Recognition; Pattern Recognition and Graphics; Production methods; Reconstruction; Shading; Vision; Vision systems; Surface detail; Sums of Gaussian; Performance capture
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-016-0979-1

This paper presents a novel approach to recover true fine surface detail of deforming meshes reconstructed from multi-view video. Template-based methods for performance capture usually produce a coarse-to-medium scale detail 4D surface reconstruction which does not contain the real high-frequency geometric detail present in the original video footage. Fine scale deformation is often incorporated in a second pass by using stereo constraints, features, or shading-based refinement. In this paper, we propose an alternative solution to this second stage by formulating dense dynamic surface reconstruction as a global optimization problem of the densely deforming surface. Our main contribution is an implicit representation of a deformable mesh that uses a set of Gaussian functions on the surface to represent the initial coarse mesh, and a set of Gaussians for the images to represent the original captured multi-view images. We effectively find the fine scale deformations for all mesh vertices, which maximize photo-temporal-consistency, by densely optimizing our model-to-image consistency energy on all vertex positions. Our formulation yields a smooth closed form energy with implicit occlusion handling and analytic derivatives. Furthermore, it does not require error-prone correspondence finding or discrete sampling of surface displacement values. We demonstrate our approach on a variety of datasets of human subjects wearing loose clothing and performing different motions. We qualitatively and quantitatively demonstrate that our technique successfully reproduces finer detail than the input baseline geometry.