The Gaze Dialogue Model: Nonverbal Communication in HHI and HRI

• Published in: IEEE transactions on cybernetics Vol. 54; no. 4; pp. 2026 - 2039
• Main Authors: Rakovic, Mirko, Ferreira Duarte, Nuno, Marques, Jorge, Billard, Aude, Santos-Victor, Jose
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.04.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Bio-inspired robotics; Communication; Control systems design; Eye Movements; Fixation, Ocular; Human in the loop; Human-robot interaction; human-robot interaction (HRI); Humanoid; Humans; Information retrieval; Interpersonal Relations; Nonverbal Communication; Predictive models; Robot kinematics; Robots; Visual tasks
• ISSN: 2168-2267; 2168-2275; 2168-2275
• DOI: 10.1109/TCYB.2022.3222077

When humans interact with each other, eye gaze movements have to support motor control as well as communication. On the one hand, we need to fixate the task goal to retrieve visual information required for safe and precise action-execution. On the other hand, gaze movements fulfil the purpose of communication, both for reading the intention of our interaction partners, as well as to signal our action intentions to others. We study this Gaze Dialogue between two participants working on a collaborative task involving two types of actions: 1) individual action and 2) action-in-interaction. We recorded the eye-gaze data of both participants during the interaction sessions in order to build a computational model, the Gaze Dialogue, encoding the interplay of the eye movements during the dyadic interaction. The model also captures the correlation between the different gaze fixation points and the nature of the action. This knowledge is used to infer the type of action performed by an individual. We validated the model against the recorded eye-gaze behavior of one subject, taking the eye-gaze behavior of the other subject as the input. Finally, we used the model to design a humanoid robot controller that provides interpersonal gaze coordination in human-robot interaction scenarios. During the interaction, the robot is able to: 1) adequately infer the human action from gaze cues; 2) adjust its gaze fixation according to the human eye-gaze behavior; and 3) signal nonverbal cues that correlate with the robot's own action intentions.