Predicting visual difference maps for computer‐generated images by integrating human visual system model and deep learning

• Published in: IET image processing Vol. 17; no. 3; pp. 901 - 915
• Main Authors: Li, Ling, Chen, Chunyi, Peng, Jun, Zhang, Ripei
• Format: Journal Article
• Language: English
• Published: Wiley 01.02.2023
• Subjects: distortion visibility; human visual perception; image quality; visual metric
• ISSN: 1751-9659; 1751-9667
• DOI: 10.1049/ipr2.12681

The quality of images generated by computer graphics rendering algorithms is mainly affected by visible distortion at some pixel locations. Image quality assessment (IQA) metrics are commonly utilized to assess the quality of rendered images, but their results are a global difference value, which does not provide pixel‐wise differences to optimize the renderings. In contrast, visibility difference models including visual perception models and deep learning models can calculate pixel‐wise visibility difference between distorted images and reference images. However, they either are only applied to a single type of visible distortion or are seriously dependent on datasets. To this end, the authors propose a novel model, dubbed Human Visual Perception and Deep Learning Image Difference Metric (HPDL‐IDM), which combines the Human Visual System (HVS) model and deep learning. HPDL‐IDM primarily consists of two modules: (i) the visual perception feature calculation module, which calculates difference maps between various kinds of features extracted from the reference image and the distorted image according to the visual characteristics of human eyes and concatenates them, and (ii) the deep learning module, which utilizes a neural network of encoder–decoder structure to train on the LocvisVC and VisTexRes datasets whose input and output are these concatenated feature difference maps and the final image distortion visibility difference map respectively. Additionally, the authors pool the final difference map into a global difference value between 0 and 1 to apply their model to many image processing tasks related to Image quality metrics (IQMs). Experimental results show that HPDL‐IDM's generalization capacity and accuracy are improved by a large margin compared to other models. This paper proposed a novel model, Human Visual Perception and Deep Learning Image Difference Metric (HPDL‐IDM), for visibility difference prediction. Unlike other models that take as input images directly, HPDL‐IDM first extracts spatial features of images and then predicts the difference. HPDL‐IDM improves the prediction accuracy for different types of distortion and effectively solves the problem that previous models can only be applied to a single type of distortion. To our knowledge, HPDL‐IDM is the first model combining image spatial feature extraction with deep learning.