Adaptation and Attention for Neural Video Coding

• Published in: 2021 IEEE International Symposium on Multimedia (ISM) pp. 240 - 244
• Main Authors: Zou, Nannan, Zhang, Honglei, Cricri, Francesco, Youvalari, Ramin G., Tavakoli, Hamed R., Lainema, Jani, Aksu, Emre, Hannuksela, Miska, Rahtu, Esa
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.01.2021
• Subjects: Adaptation models; content-adaptive; finetuning; Image coding; learned video codec; Motion estimation; Neural networks; overfitting; split attention; Training; Video coding
• DOI: 10.1109/ISM52913.2021.00047

Neural image coding represents now the state-of-the-art image compression approach. However, a lot of work is still to be done in the video domain. In this work, we propose an end-to-end learned video codec that introduces several architectural novelties as well as training novelties, revolving around the concepts of adaptation and attention. Our codec is organized as an intra-frame codec paired with an inter-frame codec. As one architectural novelty, we propose to train the inter-frame codec model to adapt the motion estimation process based on the resolution of the input video. A second architectural novelty is a new neural block that combines concepts from split-attention based neural networks and from DenseNets. Finally, we propose to overfit a set of decoder-side multiplicative parameters at inference time. Through ablation studies and comparisons to prior art, we show the benefits of our proposed techniques in terms of coding gains. We compare our codec to VVC/H.266 and RLVC, which represent the state-of-the-art traditional and end-to-end learned codecs, respectively, and to the top performing end-to-end learned approach in 2021 CLIC competition, E2E_T_OL. Our codec clearly outperforms E2E_T_OL, and compare favorably to VVC and RLVC in some settings.