Robot System Assistant (RoSA): Towards Intuitive Multi-Modal and Multi-Device Human-Robot Interaction

• Published in: Sensors (Basel, Switzerland) Vol. 22; no. 3; p. 923
• Main Authors: Strazdas, Dominykas, Hintz, Jan, Khalifa, Aly, Abdelrahman, Ahmed A., Hempel, Thorsten, Al-Hamadi, Ayoub
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 25.01.2022; MDPI
• Subjects: activity recognition; Artificial intelligence; augmented reality; Collaboration; Communication; Comparative analysis; cooperative systems; facial recognition; gesture recognition; Gestures; human-robot interaction; Humans; Middleware; Robotics; Robots; Rosa; Software; Speech; Work stations; Germany; human-robot interaction; gesture recognition; speech recognition; robot control; interactive systems; augmented reality; cooperative systems; facial recognition; activity recognition
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s22030923

This paper presents an implementation of RoSA, a Robot System Assistant, for safe and intuitive human-machine interaction. The interaction modalities were chosen and previously reviewed using a Wizard of Oz study emphasizing a strong propensity for speech and pointing gestures. Based on these findings, we design and implement a new multi-modal system for contactless human-machine interaction based on speech, facial, and gesture recognition. We evaluate our proposed system in an extensive study with multiple subjects to examine the user experience and interaction efficiency. It reports that our method achieves similar usability scores compared to the entirely human remote-controlled robot interaction in our Wizard of Oz study. Furthermore, our framework’s implementation is based on the Robot Operating System (ROS), allowing modularity and extendability for our multi-device and multi-user method.