Convolutional Neural Network-Based Block Up-Sampling for Intra Frame Coding

• Published in: IEEE transactions on circuits and systems for video technology Vol. 28; no. 9; pp. 2316 - 2330
• Main Authors: Li, Yue, Liu, Dong, Li, Houqiang, Li, Li, Wu, Feng, Zhang, Hong, Yang, Haitao
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.09.2018; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Anchors; Artificial neural networks; Block codes; Convolutional codes; Convolutional neural network (CNN); down-sampling; Encoding; Feature maps; high efficiency video coding (HEVC); Image coding; Image reconstruction; Image resolution; intra frame coding; Neural networks; Reduction; Sampling methods; Signal resolution; up-sampling; Video coding; Video compression
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2017.2727682

Inspired by the recent advances of image super-resolution using convolutional neural network (CNN), we propose a CNN-based block up-sampling scheme for intra frame coding. A block can be down-sampled before being compressed by normal intra coding, and then up-sampled to its original resolution. Different from previous studies on down/up-sampling-based coding, the up-sampling methods in our scheme have been designed by training CNN instead of hand-crafted. We explore a new CNN structure for up-sampling, which features deconvolution of feature maps, multi-scale fusion, and residue learning, making the network both compact and efficient. We also design different networks for the up-sampling of luma and chroma components, respectively, where the chroma up-sampling CNN utilizes the luma information to boost its performance. In addition, we design a two-stage up-sampling process, the first stage being within the block-by-block coding loop, and the second stage being performed on the entire frame, so as to refine block boundaries. We also empirically study how to set the coding parameters of down-sampled blocks for pursuing the frame-level rate-distortion optimization. Our proposed scheme is implemented into the high-efficiency video coding (HEVC) reference software, and a comprehensive set of experiments have been performed to evaluate our methods. Experimental results show that our scheme achieves significant bits saving compared with the HEVC anchor, especially at low bit rates, leading to on average 5.5% BD-rate reduction on common test sequences and on average 9.0% BD-rate reduction on ultrahigh definition test sequences.