Information-Theoretic Detection of Bimanual Interactions for Dual-Arm Robot Plan Generation

• Published in: IEEE robotics and automation letters Vol. 10; no. 5; pp. 4532 - 4539
• Main Authors: Merlo, Elena, Lagomarsino, Marta, Ajoudani, Arash
• Format: Journal Article
• Language: English
• Published: IEEE 01.05.2025
• Subjects: Bimanual manipulation; Data mining; Decision making; Hands; learning from demonstration; Manipulators; Robot kinematics; Robot programming; Robot sensing systems; Robots; semantic scene understanding; Topology; Trajectory
• ISSN: 2377-3766; 2377-3766
• DOI: 10.1109/LRA.2025.3552216

Programming by demonstration is a strategy to simplify the robot programming process for non-experts via human demonstrations. However, its adoption for bimanual tasks is an underexplored problem due to the complexity of hand coordination, which also hinders data recording. This letter presents a novel one-shot method for processing a single RGB video of a bimanual task demonstration to generate an execution plan for a dual-arm robotic system. To detect hand coordination policies, we apply Shannon's information theory to analyze the information flow between scene elements and leverage scene graph properties. The generated plan is a modular behavior tree that assumes different structures based on the desired arms coordination. We validated the effectiveness of this framework through multiple subject video demonstrations, which we collected and made open-source, and exploiting data from an external, publicly available dataset. Comparisons with existing methods revealed significant improvements in generating a centralized execution plan for coordinating two-arm systems.