Human hand motion prediction based on feature grouping and deep learning: Pipe skid maintenance example

• Published in: Automation in construction Vol. 138; p. 104232
• Main Authors: Zhou, Tianyu, Wang, Yubo, Zhu, Qi, Du, Jing
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 01.06.2022; Elsevier BV
• Subjects: Algorithms; Clusters; Collaboration; Construction; Deep learning; Dynamic time warping; Human behavior; Human motion; Human motion prediction; Human-robot collaboration; K-means; LightGBM; Machine learning; Maintenance; Pipes; Prediction models; Robots; Segments; Situational awareness; Symbolic aggregate approximation; Trajectories; Deep learning; K-means; Dynamic time warping; LightGBM; Human motion prediction; Symbolic aggregate approximation; Human-robot collaboration
• ISSN: 0926-5805; 1872-7891
• DOI: 10.1016/j.autcon.2022.104232

Human-robot collaboration has gained popularity in various construction applications. The key to a successful human-robot collaboration at the same construction workplace is the delicate algorithm for predicting human motions to strengthen the robot's situational awareness, i.e., robot-human awareness. Most existing approaches have focused on predicting human motions based on repetitive patterns of human behaviors in well-defined task contexts, such as specific object picking tasks, for a relatively short period of time. These methods can hardly capture the “pattern inconsistency” of human actions, i.e., the differences across people in terms of motion features and even for the same person at different time points of the task. This paper proposes an analytical pipeline that segments and clusters the human inconsistent behaviors into different pattern groups and builds separate human motion pattern prediction models correspondingly. The proposed method, Human Motion Feature Grouping and Prediction (HMFGP), quantifies the spatiotemporal relationship between gaze focus and hand movement trajectories, segments the raw data based on the detected gaze-hand relationship pattern changes, and clusters the matched gaze-hand data segments into several pattern groups based on the pattern similarity of the gaze-hand relationships. Then a time series Deep Learning method is used to predict hand motions based on gaze focus trajectories for each of the pattern groups. The gaze and hand motion data of a human subject experiment (n = 120) for pipe skid maintenance was used to test the prediction performance of HMFGP. The result shows that HMFGP can significantly improve the accuracy of human hand motion prediction and help quantity different patterns of human motions for specific analyses. •This paper proposes a deep learning method for predicting hand motions based on gaze focus.•Human subject experiment (n = 120) was conducted in VR to collect data in a pipe skid task.•Behavioral inconsistency is quantified and used as basis for separate machine learning training.•Results show that the proposed method reduces prediction errors by 16.96%.•The findings suggest the importance of mining pattern inconsistency in human motion prediction.