NHK Meta Studio: A Compact Volumetric TV Studio for 3-D Reconstruction

• Published in: IEEE transactions on broadcasting Vol. 69; no. 1; pp. 2 - 9
• Main Authors: Morioka, Hirofumi, Misu, Toshie, Suginoshita, Taichi, Mitsumine, Hideki
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.03.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3D reconstruction; Artificial neural networks; Calibration; camera placement; Cameras; Image reconstruction; Keying; Solid modeling; Surface reconstruction; Three dimensional models; Three-dimensional displays; TV studio; video compensation; volumetric studio
• ISSN: 0018-9316; 1557-9611
• DOI: 10.1109/TBC.2022.3210367

Many studies have been conducted on the 3D reconstruction of subjects using multiple fixed cameras. Accepting the trade-off between the number of cameras and reconstruction quality, our studio is designed to capture high-quality models of one or two subjects for TV program use. Several cameras are mounted on a hemispherical dome with the stage in the center and a cloth cover on the frame for chroma-keying. The optimal camera numbers and placements for reconstruction were determined by simulation, and the 3D reconstruction was performed as a point cloud by a combination of visual hull and stereo matching. The quality was still not high enough, however, so we also added a surface light field to the point cloud to obtain the weighted average of rays from camera images close to the viewpoint. In the final stage, the images were then combined to the video, and errors generated during the reconstruction were compensated for by use of a deep neural network (DNN) for video translation. An offline processing studio has been built as a preliminary step towards real-time processing, and the reconstructed 3D images have been evaluated subjectively for a number of subjects. These studies confirm the effectiveness of this studio design.