Deep Color Guided Coarse-to-Fine Convolutional Network Cascade for Depth Image Super-Resolution

• Published in: IEEE transactions on image processing Vol. 28; no. 2; pp. 994 - 1006
• Main Authors: Yang Wen, Bin Sheng, Ping Li, Weiyao Lin, Feng, David Dagan
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.02.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; coarse-to-fine convolutional neural network; Color; color guidance; Copying; Depth super-resolution; filter kernel learning; Fuses; High resolution; Image color analysis; Image edge detection; Image filters; Image resolution; Kernel; Kernels; Sensors; Spatial resolution; State of the art
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2018.2874285

Depth image super-resolution is a significant yet challenging task. In this paper, we introduce a novel deep color guided coarse-to-fine convolutional neural network (CNN) framework to address this problem. First, we present a data-driven filter method to approximate the ideal filter for depth image super-resolution instead of hand-designed filters. Based on large data samples, the filter learned is more accurate and stable for upsampling depth image. Second, we introduce a coarse-to-fine CNN to learn different sizes of filter kernels. In the coarse stage, larger filter kernels are learned by the CNN to achieve crude high-resolution depth image. As to the fine stage, the crude high-resolution depth image is used as the input so that smaller filter kernels are learned to gain more accurate results. Benefit from this network, we can progressively recover the high frequency details. Third, we construct a color guidance strategy that fuses color difference and spatial distance for depth image upsampling. We revise the interpolated high-resolution depth image according to the corresponding pixels in high-resolution color maps. Guided by color information, the depth of high-resolution image obtained can alleviate texture copying artifacts and preserve edge details effectively. Quantitative and qualitative experimental results demonstrate our state-of-the-art performance for depth map super-resolution.