Detection of eye contact with deep neural networks is as accurate as human experts

• Published in: Nature communications Vol. 11; no. 1; pp. 6386 - 10
• Main Authors: Chong, Eunji, Clark-Whitney, Elysha, Southerland, Audrey, Stubbs, Elizabeth, Miller, Chanel, Ajodan, Eliana L., Silverman, Melanie R., Lord, Catherine, Rozga, Agata, Jones, Rebecca M., Rehg, James M.
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 14.12.2020; Nature Publishing Group; Nature Portfolio
• Subjects: 639/705/117; 706/689/477; Artificial neural networks; Autism; Autism Spectrum Disorder; Child, Preschool; Coders; Communication; Communication skills; Deep Learning; Diagnosis; Eye; Eye contact; Female; Hand; Humanities and Social Sciences; Humans; Infant; Machine Learning; Male; Models, Theoretical; multidisciplinary; Neural networks; Neural Networks, Computer; Recall; Science; Science (multidisciplinary); Social behavior
• DOI: 10.1038/s41467-020-19712-x
• ISSN: 2041-1723

Eye contact is among the most primary means of social communication used by humans. Quantification of eye contact is valuable as a part of the analysis of social roles and communication skills, and for clinical screening. Estimating a subject’s looking direction is a challenging task, but eye contact can be effectively captured by a wearable point-of-view camera which provides a unique viewpoint. While moments of eye contact from this viewpoint can be hand-coded, such a process tends to be laborious and subjective. In this work, we develop a deep neural network model to automatically detect eye contact in egocentric video. It is the first to achieve accuracy equivalent to that of human experts. We train a deep convolutional network using a dataset of 4,339,879 annotated images, consisting of 103 subjects with diverse demographic backgrounds. 57 subjects have a diagnosis of Autism Spectrum Disorder. The network achieves overall precision of 0.936 and recall of 0.943 on 18 validation subjects, and its performance is on par with 10 trained human coders with a mean precision 0.918 and recall 0.946. Our method will be instrumental in gaze behavior analysis by serving as a scalable, objective, and accessible tool for clinicians and researchers. Eye contact is a key social behavior and its measurement could facilitate the diagnosis and treatment of autism. Here the authors show that a deep neural network model can detect eye contact as accurately has human experts.