A Database and Model for the Visual Quality Assessment of Super-Resolution Videos

• Published in: IEEE transactions on broadcasting Vol. 70; no. 2; pp. 516 - 532
• Main Authors: Zhou, Fei, Sheng, Wei, Lu, Zitao, Qiu, Guoping
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.06.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Artificial neural networks; Data analysis; Display devices; Evaluation; Feature extraction; Quality assessment; Quality assurance; Quality enhancement assessment; Spatial resolution; super-resolution video quality assessment; Superresolution; Video; video database; Videos; Viewing; Visualization
• ISSN: 0018-9316; 1557-9611
• DOI: 10.1109/TBC.2024.3382949

Video super-resolution (SR) has important real world applications such as enhancing viewing experiences of legacy low-resolution videos on high resolution display devices. However, there are no visual quality assessment (VQA) models specifically designed for evaluating SR videos while such models are crucially important both for advancing video SR algorithms and for viewing quality assurance. This paper addresses this gap. We start by contributing the first video super-resolution quality assessment database (VSR-QAD) which contains 2,260 SR videos annotated with mean opinion score (MOS) labels collected through an approximately 400 man-hours psychovisual experiment by a total of 190 subjects. We then build on the new VSR-QAD and develop the first VQA model specifically designed for evaluating SR videos. The model features a two-stream convolutional neural network architecture and a two-stage training algorithm designed for extracting spatial and temporal features characterizing the quality of SR videos. We present experimental results and data analysis to demonstrate the high data quality of VSR-QAD and the effectiveness of the new VQA model for measuring the visual quality of SR videos. The new database and the code of the proposed model will be available online at https://github.com/key1cdc/VSRQAD .