Neurobehavioral assessment of force feedback simulation in industrial robotic teleoperation

• Published in: Automation in construction Vol. 126; p. 103674
• Main Authors: Zhu, Qi, Du, Jing, Shi, Yangming, Wei, Paul
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 01.06.2021; Elsevier BV
• Subjects: Construction industry; Control systems; Eye movements; Feedback; fNIRS; Force feedback; Human performance; Human-robot collaboration; Industrial robots; Infrared analysis; Infrared spectra; Infrared tracking; Man-machine interfaces; Medical imaging; Near infrared radiation; Neurophysiological analysis; Operators; Simulation; Teleoperation; Telerobotics; Virtual reality; Workplaces; Teleoperation; Neurophysiological analysis; fNIRS; Force feedback; Human-robot collaboration; Virtual reality
• ISSN: 0926-5805; 1872-7891
• DOI: 10.1016/j.autcon.2021.103674

Telerobotic operation, i.e., a human operator to manipulate remote robotic systems at a distance, has started to gain its popularity in the construction industry. It is expected to help tackle operational challenges in dynamic construction workplaces. The success of telerobotic operation builds on the effective design of the human-robot interface to provide human operators with necessary senses about the remote workplaces, involving multimodal sensory cues, such as visual, audio and haptic feedback. Especially the force feedback design in telerobotic control interface is of central interest and is becoming the main feature of the bilateral control system for teleoperation, as it helps provide feedback about heavy physical interactions and processes in typical construction operations. Nonetheless, how force feedback simulation solutions affect the human operator's perceptional and behavioral reactions is less understood. This paper investigates the neurobehavioral performance of operators with a bilateral control system in a typical industrial valve operation experiment (n = 21). The experiment tested two force feedback conditions: Realistic (the system replicates the exact same feeling of the torque in valve manipulation operations) and Mediated (the simulation reduces the force on the human operator end by 50% to enable more flexible controls). The performance of the participants was evaluated via various metrics, including task performance, human performance and operational velocity uniformity. Data was collected with eye-tracking, neuroimaging (functional near-infrared spectroscopy, fNIRS), motion analysis, and NASA TLX surveys. The results indicated that the mediated force feedback in bilateral telerobotic operation helped more accurate operation, increased dual tasking, reduced cognitive load and more efficient neural functions; yet it encouraged participants to engage in more irregular actions, showing as dramatic changes in valve rotating speeds. The findings suggest that the force feedback design of telerobotic systems should be more carefully thought through to balance the advantages and disadvantages. •This paper tests neurobehavioral reactions to force feedback simulation in telerobotic controls.•Human subject experiment (n = 30) was conducted in VR to collect fNIRS and behavioral data.•Results show mediated force feedback improved task performance and human performance.•Mediated force feedback encouraged more irregular operational performance.•The findings suggest a more delicate design of human-robot interface in telerobotic operation.