Spatio-Temporally Consistent View Synthesis From Video-Plus-Depth Data With Global Optimization

• Published in: IEEE transactions on circuits and systems for video technology Vol. 24; no. 1; pp. 74 - 84
• Main Authors: HSU, Hsiao-An, CHIANG, Chen-Kuo, LAI, Shang-Hong
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.01.2014; Institute of Electrical and Electronics Engineers
• Subjects: Applied sciences; Color; Detection, estimation, filtering, equalization, prediction; Exact sciences and technology; Image color analysis; Image processing; Information, signal and communications theory; Integrated optics; Labeling; Markov random field (MRF); Minimization; multilevel belief propagation (BP); Optical imaging; Optimization; Signal and communications theory; Signal processing; Signal, noise; spatio-temporally consistency; Telecommunications and information theory; video-plus-depth; view synthesis; Performance evaluation; Image processing; Energy function; Warping; Video signal; Image reconstruction; Random field; Credal approach; Image sequence; view synthesis; Multilevel system; Probabilistic approach; Motion estimation; Virtual image; Global optimum; Markov random field (MRF); Markov model; Algorithm; Computational complexity; multilevel belief propagation (BP); Signal processing; spatio-temporally consistency; video-plus-depth; Tridimensional image; Signal synthesis
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2013.2276699

We propose a novel algorithm to generate a virtual-view video from a video-plus-depth sequence. The proposed method enforces the spatial and temporal consistency in the disocclusion regions by formulating the problem as an energy minimization problem in a Markov random field (MRF) framework. At the system level, we first recover the depth images and the motion vector maps after the image warping with the preprocessed depth map. Then, we formulate the energy function for the MRF with additional shift variables for each node. To reduce the high computational complexity of applying belief propagation (BP) to this problem, we present a multilevel BPs by using BP with smaller numbers of label candidates for each level. Finally, the Poisson image reconstruction is applied to improve the color consistency along the boundary of the disocclusion region in the synthesized image. Experimental results demonstrate the performance of the proposed method on several publicly available datasets.