Video-Based Eye Blink Identification and Classification

• Published in: IEEE journal of biomedical and health informatics Vol. 26; no. 7; pp. 3284 - 3293
• Main Authors: Nousias, George, Panagiotopoulou, Eirini-Kanella, Delibasis, Konstantinos, Chaliasou, Aikaterini-Maria, Tzounakou, Anastasia-Maria, Labiris, Georgios
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.07.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Classification; Coders; Complete and incomplete blink classification; Deep learning; Diseases; Encoders-Decoders; eye blink detection; Eyelid; Eyelids; Faces; Image acquisition; Image resolution; Image segmentation; Iris; iris and eyelid segmentation; Machine learning; Mathematical analysis; Training; video analysis
• ISSN: 2168-2194; 2168-2208; 2168-2208
• DOI: 10.1109/JBHI.2022.3153407

Blink detection and classification can provide a very useful clinical indicator, because of its relation with many neurological and ophthalmological conditions. In this work, we propose a system that automatically detects and classifies blinks as " complete " or " incomplete " in high resolution image sequences zoomed into the participants' face, acquired during clinical examination using near-Infrared illumination. This method utilizes state-of-the-art (DeepLabv3+) deep learning encoder-decoder neural architecture -DLED to segment iris and eyelid in both eyes in the acquired images. The sequence of the segmented frames is post-processed to calculate the distance between the eyelids of each eye (palpebral fissure height) and the corresponding iris diameter. These quantities are temporally filtered and their fraction is subject to adaptive thresholding to identify blinks and determine their type, independently for each eye. The proposed system was tested on 15 participants, each with one video of 4 to 10 minutes. Several metrics of blink detection and classification accuracy were calculated against the ground truth, which was generated by three (3) independent experts, whose conflicts were resolved by a senior expert. Results show that the proposed system achieved F1-score 95.3% and 80.9% for the classification of complete and incomplete blinks respectively, collectively for all 15 participants, outperforming all 3 experts. The proposed system was proven robust in handling unexpected participant movements and actions, as well as glare and reflections from the spectacles, or face obstruction by facemasks.