Gest-SAR: A Gesture-Controlled Spatial AR System for Interactive Manual Assembly Guidance with Real-Time Operational Feedback

• Published in: Machines (Basel) Vol. 13; no. 8; p. 658
• Main Authors: Hasan, Naimul, Alkan, Bugra
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 27.07.2025
• Subjects: Accuracy; Adaptability; Adaptive technology; Augmented reality; Automation; Closed loops; Collaboration; Cycle time; Efficiency; Ergonomics; Error correction & detection; Error feedback; Feedback; Gesture recognition; Intelligent manufacturing systems; Interactive control; Interactive systems; Manufacturing; Order picking; Product lines; Productivity; Real time operation; Robots; User satisfaction; Workers; Workload; Workloads
• ISSN: 2075-1702; 2075-1702
• DOI: 10.3390/machines13080658

Manual assembly remains essential in modern manufacturing, yet the increasing complexity of customised production imposes significant cognitive burdens and error rates on workers. Existing Spatial Augmented Reality (SAR) systems often operate passively, lacking adaptive interaction, real-time feedback and a control system with gesture. In response, we present Gest-SAR, a SAR framework that integrates a custom MediaPipe-based gesture classification model to deliver adaptive light-guided pick-to-place assembly instructions and real-time error feedback within a closed-loop interaction instance. In a within-subject study, ten participants completed standardised Duplo-based assembly tasks using Gest-SAR, paper-based manuals, and tablet-based instructions; performance was evaluated via assembly cycle time, selection and placement error rates, cognitive workload assessed by NASA-TLX, and usability test by post-experimental questionnaires. Quantitative results demonstrate that Gest-SAR significantly reduces cycle times with an average of 3.95 min compared to Paper (Mean = 7.89 min, p < 0.01) and Tablet (Mean = 6.99 min, p < 0.01). It also achieved 7 times less average error rates while lowering perceived cognitive workload (p < 0.05 for mental demand) compared to conventional modalities. In total, 90% of the users agreed to prefer SAR over paper and tablet modalities. These outcomes indicate that natural hand-gesture interaction coupled with real-time visual feedback enhances both the efficiency and accuracy of manual assembly. By embedding AI-driven gesture recognition and AR projection into a human-centric assistance system, Gest-SAR advances the collaborative interplay between humans and machines, aligning with Industry 5.0 objectives of resilient, sustainable, and intelligent manufacturing.