Autonomous Teleoperated Robotic Arm Based on Imitation Learning Using Instance Segmentation and Haptics Information

• Published in: Journal of advanced computational intelligence and intelligent informatics Vol. 29; no. 1; pp. 79 - 94
• Main Authors: Imai, Kota, Takahashi, Yasutake, Tsuichihara, Satoki, Haruna, Masaki
• Format: Journal Article
• Language: English
• Published: Tokyo Fuji Technology Press Co. Ltd 20.01.2025
• Subjects: Grasping force; Haptics; Image segmentation; Instance segmentation; Robot arms; Robot learning; Robots
• ISSN: 1343-0130; 1883-8014
• DOI: 10.20965/jaciii.2025.p0079

Teleoperated robots are attracting attention as a solution to the pressing labor shortage. To reduce the burden on the operators of teleoperated robots and improve manpower efficiency, research is underway to make these robots more autonomous. However, end-to-end imitation learning models that directly map camera images to actions are vulnerable to changes in image background and lighting conditions. To improve robustness against these changes, we modified the learning model to handle segmented images where only the arm and the object are preserved. The task success rate for the demonstration data and the environment with different backgrounds was 0.0% for the model with the raw image input and 66.0% for the proposed model with segmented image input, with the latter having achieved a significant improvement. However, the grasping force of this model was stronger than that during the demonstration. Accordingly, we added haptics information to the observation input of the model. Experimental results show that this can reduce the grasping force.