An Active Task Cognition Method for Home Service Robot Using Multi-Graph Attention Fusion Mechanism

• Published in: IEEE transactions on circuits and systems for video technology Vol. 34; no. 6; pp. 4957 - 4972
• Main Authors: Cui, Yongcheng, Tian, Guohui, Jiang, Zhengsong, Zhang, Mengyang, Gu, Yu, Wang, Yifei
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.06.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Active task cognition; Cognition; Encoders-Decoders; Feature extraction; graph-based reasoning; Multitasking; robotic visual cognition; Robots; Scene analysis; scene understanding; Semantics; Service robots; Task analysis; Visualization
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2023.3339292

Active Task Cognition (ATC) requires the robot to comprehend the current scene using the image within the field of view, enabling them to reason about appropriate and executable tasks, thus allowing the robot to achieve service task scene discovery capability similar to humans. This capability is paramount for robots to provide comfort and intelligent service while performing their tasks. To enhance home service robots' ATC capability, a multi-graph fusion mechanism based on Graph Attention Network (GAT) is proposed in this paper to model the semantic feature related to the task. First, a multi-graph fusion encoder is proposed to maximally capture the integrated features of objects, tasks, and scenes from the images, thereby obtaining a semantic representation related to the home service task from the robot's perspective. Next, to enhance the interpretability of the model, we propose a multi-task scene understanding decoder based on the attention mechanism to utilize the integration features of multi-graph fusion efficiently. Lastly, we present a loss function for multi-task scene understanding in the proposed Encoder-Decoder network model for scene comprehension. Furthermore, a new dataset comprising various daily household tasks is constructed in the experiments. Extensive experimental results indicate that the proposed method significantly enhances the robot's active cognitive abilities in service tasks, empowering it with advanced levels of intelligence.