Texture-based Error Analysis for Image Super-Resolution

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) Vol. 2022; pp. 2108 - 2117
• Main Authors: Magid, Salma Abdel, Lin, Zudi, Wei, Donglai, Zhang, Yulun, Gu, Jinjin, Pfister, Hanspeter
• Format: Conference Proceeding Journal Article
• Language: English
• Published: United States IEEE 01.06.2022
• Subjects: Biological system modeling; Computational modeling; Error analysis; Low-level vision; Datasets and evaluation; Explainable computer vision; Vision applications and systems; Measurement; Pattern recognition; Semantics; Superresolution
• ISSN: 1063-6919; 1063-6919; 2575-7075
• DOI: 10.1109/CVPR52688.2022.00216

Evaluation practices for image super-resolution (SR) use a single-value metric, the PSNR or SSIM, to determine model performance. This provides little insight into the source of errors and model behavior. Therefore, it is beneficial to move beyond the conventional approach and reconceptualize evaluation with interpretability as our main priority. We focus on a thorough error analysis from a variety of perspectives. Our key contribution is to leverage a texture classifier, which enables us to assign patches with semantic labels, to identify the source of SR errors both globally and locally. We then use this to determine (a) the semantic alignment of SR datasets, (b) how SR models perform on each label, (c) to what extent high-resolution (HR) and SR patches semantically correspond, and more. Through these different angles, we are able to highlight potential pitfalls and blindspots. Our overall investigation highlights numerous unexpected insights. We hope this work serves as an initial step for debugging blackbox SR networks.