Interventionalist Hand Motion Recognition with Convolutional Neural Network in Robot-assisted Coronary Interventions

• Published in: IEEE sensors journal Vol. 23; no. 15; p. 1
• Main Authors: Akinyemi, Toluwanimi O., Omisore, Olatunji M., Du, Wenjing, Duan, Wenke, Chen, Xingyu, Yi, Guanlin, Wang, Lei
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.08.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Artificial neural networks; Cardiology; Catheters; Convolutional neural network; Deep learning; Feature maps; hand motion classification; Hidden Markov models; interventionalist; Machine learning; Motion perception; Neural networks; percutaneous coronary interventions; Real time; Recognition; Robot dynamics; Robot sensing systems; robotic catheterization; Robotic surgery; Robots; sensor-based data acquisition; Sensors; Sensory gloves; signal processing; Solid modeling; Training
• ISSN: 1530-437X; 1558-1748
• DOI: 10.1109/JSEN.2023.3281009

The development of procedural competencies by interventional cardiology fellows for minimally invasive treatment of cardiac diseases depend on their effective training and evaluation. Learning tool manipulation for safe robotic percutaneous coronary interventions requires expert supervision and use of high-fidelity systems. However, with limited proficiency for real-time hand motion recognition. Therefore, this study proposes a deep learning-based model for identifying operators' actions. The proposed model is based on the convolutional neural network that consist of one dimensional convolutional layers for automatic feature map extraction, down sampling, and fully connected layers for inference. The developed models were evaluated using a multi-modal dataset collected from sensory glove, EM, and sEMG sensors and real-time angiograms in the course of in-vivo catheterization trials performed by nine subjects (two experts and seven novices) using a custom-built robotic catheter system. The results indicate that the model achieves a 92-96% accuracy in identifying five actions across four clusters compared with a recognition performance of 83% when recognizing all six actions. Furthermore, we compared the proposed model performance with existing studies, the analyses show that our model has a 2-3% higher accuracy for five-action recognition. Therefore, the proposed model could be employed for real-time hand motion recognition in R-PCI trials.