Markerless Shape and Motion Capture From Multiview Video Sequences

• Published in: IEEE transactions on circuits and systems for video technology Vol. 21; no. 3; pp. 320 - 334
• Main Authors: Li, Kun, Dai, Qionghai, Xu, Wenli
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.03.2011; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3-D mesh; Applied sciences; Circuits; Clouds; Computational modeling; deformation; depth map; Exact sciences and technology; Finite element method; Geometry; Human; Image processing; Image reconstruction; Information, signal and communications theory; Markers; Mathematical models; Merging; Miscellaneous; motion capture; Pattern recognition; Pixel; Reconstruction; Representations; Shape; shape recovery; Signal and communications theory; Signal processing; Signal representation. Spectral analysis; Signal, noise; Solid modeling; Studies; Telecommunications and information theory; Three dimensional displays; Triangulation; Deformation; Clouds; Image processing; Video signal; Multiple image; 3-D mesh; Image analysis; Accuracy; shape recovery; Image sequence; Scene analysis; Signal processing; depth map; Motion detection; Pattern recovery; motion capture
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2011.2106251

We propose a new markerless shape and motion capture approach from multiview video sequences. The shape recovery method consists of two steps: separating and merging. In the separating step, the depth map represented with a point cloud for each view is generated by solving a proposed variational model, which is regularized by four constraints to ensure the accuracy and completeness of the reconstruction. Then, in the merging step, the point clouds of all the views are merged together and reconstructed into a 3-D mesh using a marching cubes method with silhouette constraints. Experiments show that the geometric details are faithfully preserved in each estimated depth map. The 3-D meshes reconstructed from the estimated depth maps are watertight and present rich geometric details, even for non-convex objects. Taking the reconstructed 3-D mesh as the underlying scene representation, a volumetric deformation method with a new positional-constraint computation scheme is proposed to automatically capture motions of the 3-D object. Our method can capture non-rigid motions even for loosely dressed humans without the aid of markers.