Estimation of the Quality of Experience During Video Streaming From Facial Expression and Gaze Direction

• Published in: IEEE eTransactions on network and service management Vol. 17; no. 4; pp. 2702 - 2716
• Main Authors: Porcu, Simone, Floris, Alessandro, Voigt-Antons, Jan-Niklas, Atzori, Luigi, Moller, Sebastian
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.12.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Blurring; Brain modeling; Classifiers; Electroencephalography; Face; Face recognition; facial expressions; Feature extraction; Feedback; gaze direction; Laboratory tests; machine learning; Muscles; QoE estimation; Quality of experience; Streaming media; Two dimensional displays; User experience; video key quality indicators; Video streaming; Video transmission
• ISSN: 1932-4537; 1932-4537
• DOI: 10.1109/TNSM.2020.3018303

This article investigates the possibility to estimate the perceived Quality of Experience (QoE) automatically and unobtrusively by analyzing the face of the consumer of video streaming services, from which facial expression and gaze direction are extracted. If effective, this would be a valuable tool for the monitoring of personal QoE during video streaming services without asking the user to provide feedback, with great advantages for service management. Additionally, this would eliminate the bias of subjective tests and would avoid bothering the viewers with questions to collect opinions and feedback. The performed analysis relies on two different experiments: i) a crowdsourcing test, where the videos are subject to impairments caused by long initial delays and re-buffering events; ii) a laboratory test, where the videos are affected by blurring effects. The facial Action Units (AU) that represent the contractions of specific facial muscles together with the position of the eyes' pupils are extracted to identify the correlation between perceived quality and facial expressions. An SVM with a quadratic kernel and a k-NN classifier have been tested to predict the QoE from these features. These have also been combined with measured application-level parameters to improve the quality prediction. From the performed experiments, it results that the best performance is obtained with the k-NN classifier by combining all the described features and after training it with both the datasets, with a prediction accuracy as high as 93.9% outperforming the state of the art achievements.