PVC-STIM: Perceptual video coding based on spatio-temporal influence map

• Published in: Signal, image and video processing Vol. 16; no. 7; pp. 1841 - 1849
• Main Authors: Wang, Bing, Gao, Pan, Peng, Qiang, Zhang, Qianwen, Wu, Xiao, Xiang, Wei
• Format: Journal Article
• Language: English
• Published: London Springer London 01.10.2022; Springer Nature B.V
• Subjects: Coding; Computer Imaging; Computer Science; Image Processing and Computer Vision; Lagrange multiplier; Multimedia Information Systems; Original Paper; Pattern Recognition and Graphics; Signal,Image and Speech Processing; Vision; Visual effects; Visual stimuli; Video coding; Human visual system; Rate distortion optimization; Perceptual video coding
• ISSN: 1863-1703; 1863-1711
• DOI: 10.1007/s11760-022-02143-0

Since the ultimate consumers and judgers of most video applications are human subjects, there has been growing interest in incorporating characteristics of the human visual system (HVS) in video coding for the further development of coding technology, called perceptual video coding (PVC). Although there have been numerous PVC methods reported in the literature to date, exploring various factors affecting the performance of PVC still remains challenging due to the complexity of HVS and its perceptual mechanisms. In this paper, we propose a perceptual video coding scheme based on a novel spatio-temporal influence map model (PVC-STIM). In the first step, we develop a novel perceptual model by considering multiple perceptual characteristics of HVS, with a special focus on the fusion of several spatial and temporal features, i.e. spatial masking effect, spatial stimuli, visual saliency and temporal motion attention. In the second step, the proposed perceptual model is incorporated into the classic video coding framework to adjust the Lagrange multiplier in order to reasonably allocate visual quality, which improves the rate and perceived distortion performance. Experimental results show that the proposed PVC-STIM method can achieve on average 8.76% bitrate savings while retaining similar perceived quality, compared to HEVC, and can also outperform two PVC approaches.