A novel approach for automatic annotation of human actions in 3D point clouds for flexible collaborative tasks with industrial robots

• Published in: Frontiers in robotics and AI Vol. 10; p. 1028329
• Main Authors: Krusche, Sebastian, Al Naser, Ibrahim, Bdiwi, Mohamad, Ihlenfeldt, Steffen
• Format: Journal Article
• Language: English
• Published: Switzerland Frontiers Media S.A 15.02.2023
• Subjects: data labeling; deep learning; human activity recognition; point cloud annotation; robotics; Robotics and AI; point cloud annotation; robotics; deep learning; data labeling; human activity recognition
• ISSN: 2296-9144; 2296-9144
• DOI: 10.3389/frobt.2023.1028329

Manual annotation for human action recognition with content semantics using 3D Point Cloud (3D-PC) in industrial environments consumes a lot of time and resources. This work aims to recognize, analyze, and model human actions to develop a framework for automatically extracting content semantics. Main Contributions of this work: 1. design a multi-layer structure of various DNN classifiers to detect and extract humans and dynamic objects using 3D-PC preciously, 2. empirical experiments with over 10 subjects for collecting datasets of human actions and activities in one industrial setting, 3. development of an intuitive GUI to verify human actions and its interaction activities with the environment, 4. design and implement a methodology for automatic sequence matching of human actions in 3D-PC. All these procedures are merged in the proposed framework and evaluated in one industrial Use-Case with flexible patch sizes. Comparing the new approach with standard methods has shown that the annotation process can be accelerated by 5.2 times through automation.