Modeling Subject Scoring Behaviors in Subjective Experiments Based on a Discrete Quality Scale

• Published in: IEEE transactions on multimedia Vol. 26; pp. 8742 - 8757
• Main Authors: Tiotsop, Lohic Fotio, Servetti, Antonio, Barkowsky, Marcus, Masala, Enrico
• Format: Journal Article
• Language: English
• Published: IEEE 2024
• Subjects: discrete quality scale; Media; Noise measurement; Numerical models; Probabilistic logic; Proposals; Recommender systems; subject bias weights; subject inconsistency; subject scoring model; Subjective quality recovery; Systematics
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2024.3382483

Several approaches have been proposed to estimate quality in subjective experiments while highlighting peculiar subject behaviors. However, there is some room for improvement in existing approaches, both in terms of robustness to noise and the ability to accurately indicate several peculiar subject behaviors in subjective experiments. This work advances the state-of-the-art in three main directions: i) A new approach to estimate the subjective quality from noisy ratings is proposed and is shown to be more robust to noise than are four state-of-the-art approaches; ii) a novel subject scoring model is proposed that makes it possible to highlight several peculiar behaviors typically observed in subjective experiments; and iii) our proposed probabilistic subject scoring model results from the proof of a theorem, whereas in previous approaches a probabilistic scoring model is assumed a priori . This represents an important first step toward models supported by a stronger theoretical foundation. Numerical experiments conducted on several datasets highlight the effectiveness of our proposal.