A New Model for Assembly Task Recognition: A Case Study of Seru Production System

• Published in: IEEE access p. 1
• Main Authors: Torkul, Orhan, Selvi, Ihsan Hakan, Sisci, Merve, Diren, Deniz Demircioglu
• Format: Journal Article
• Language: English
• Published: IEEE 22.10.2024
• Subjects: Assembly; assembly action recognition; Convolutional neural networks; Data augmentation; Deep learning; Feature extraction; Indexes; Logic gates; MediaPipe; pose estimation; Production; Production systems; seru production system; Training
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2024.3484955

The Seru production system is an innovative assembly system that combines the flexibility of shop floor production and the high efficiency of assembly lines. In addition to its advantages, seru-type production has the disadvantage that all the tasks required for the assembly of a product are completed in a yatai by a cross-trained worker. This results in a higher risk of production errors compared to assembly lines. In order to mitigate these risks, a real-time control system is considered necessary. Recognition of assembly tasks is needed to prevent quality defects in the Seru production system. To meet this need, in this study, a skeleton-based deep learning hybrid Convolutional Neural Network-Bi-directional Gated Recurrent Unit-Convolutional Neural Network (CNN-BiGRU-CNN) assembly task recognition model is developed. In this study, a two-stage data augmentation approach is proposed to improve the performance of the model developed by utilizing the worker skeleton data obtained with Mediapipe Holistic infrastructure from the assembly video collected from the seru production system. The effectiveness of the model was evaluated by comparing it with Convolutional Neural Network (CNN), Long Short Term Memory (LSTM), Gated Recurrent Unit (GRU), Bi-directional Long Short Term Memory (BiLSTM), BiGRU and hybrid models created by combinations of these models. The proposed data augmentation approach improved the performance of all the models compared in the study. With the proposed hybrid CNN-BiGRU-CNN model, the best performance was achieved with a prediction accuracy of 96%.