An End-to-End Learning Framework for Video Compression

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 43; no. 10; pp. 3292 - 3308
• Main Authors: Lu, Guo, Zhang, Xiaoyun, Ouyang, Wanli, Chen, Li, Gao, Zhiyong, Xu, Dong
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptive optics; Coders; Coding; end-to-end optimization; Estimation; Image coding; image compression; Motion estimation; neural network; Neural networks; Optical distortion; Optical flow (image analysis); Optical imaging; Video compression
• ISSN: 0162-8828; 1939-3539; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2020.2988453

Traditional video compression approaches build upon the hybrid coding framework with motion-compensated prediction and residual transform coding. In this paper, we propose the first end-to-end deep video compression framework to take advantage of both the classical compression architecture and the powerful non-linear representation ability of neural networks. Our framework employs pixel-wise motion information, which is learned from an optical flow network and further compressed by an auto-encoder network to save bits. The other compression components are also implemented by the well-designed networks for high efficiency. All the modules are jointly optimized by using the rate-distortion trade-off and can collaborate with each other. More importantly, the proposed deep video compression framework is very flexible and can be easily extended by using lightweight or advanced networks for higher speed or better efficiency. We also propose to introduce the adaptive quantization layer to reduce the number of parameters for variable bitrate coding. Comprehensive experimental results demonstrate the effectiveness of the proposed framework on the benchmark datasets.