Can learned frame prediction compete with block motion compensation for video coding?

• Published in: Signal, image and video processing Vol. 15; no. 2; pp. 401 - 410
• Main Authors: Sulun, Serkan, Tekalp, A. Murat
• Format: Journal Article
• Language: English
• Published: London Springer London 01.03.2021; Springer Nature B.V
• Subjects: Codec; Computer Imaging; Computer Science; Image Processing and Computer Vision; Motion compensation; Multimedia Information Systems; Original Paper; Pattern Recognition and Graphics; Signal,Image and Speech Processing; Video compression; Vision; Deep learning; Predictive frame difference; Frame prediction; HEVC-Intra codec; Rate-distortion performance
• ISSN: 1863-1703; 1863-1711
• DOI: 10.1007/s11760-020-01751-y

Given recent advances in learned video prediction, we investigate whether a simple video codec using a pretrained deep model for next frame prediction based on previously encoded/decoded frames without sending any motion side information can compete with standard video codecs based on block motion compensation. Frame differences given learned frame predictions are encoded by a standard still-image (intra) codec. Experimental results show that the rate distortion performance of the simple codec with symmetric complexity is on average better than that of x264 codec on 10 MPEG test videos, but does not yet reach the level of x265 codec. This result demonstrates the power of learned frame prediction (LFP), since unlike motion compensation, LFP does not use information from the current picture. The implications of training with ℓ 1 , ℓ 2 or combined ℓ 2 and adversarial loss on prediction performance and compression efficiency are analyzed.