SurgeoNet: Realtime 3D Pose Estimation of Articulated Surgical Instruments from Stereo Images using a Synthetically-trained Network

Surgery monitoring in Mixed Reality (MR) environments has recently received substantial focus due to its importance in image-based decisions, skill assessment, and robot-assisted surgery. Tracking hands and articulated surgical instruments is crucial for the success of these applications. Due to the...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Ahmed Tawfik Aboukhadra, Robertini, Nadia, Malik, Jameel, Elhayek, Ahmed, Reis, Gerd, Stricker, Didier
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 02.10.2024
Subjects
Mixed reality
Neural networks
Pose estimation
Real time
Robotic surgery
Surgical instruments
Synthetic data
Online AccessGet full text

Cover

More Information
Summary:Surgery monitoring in Mixed Reality (MR) environments has recently received substantial focus due to its importance in image-based decisions, skill assessment, and robot-assisted surgery. Tracking hands and articulated surgical instruments is crucial for the success of these applications. Due to the lack of annotated datasets and the complexity of the task, only a few works have addressed this problem. In this work, we present SurgeoNet, a real-time neural network pipeline to accurately detect and track surgical instruments from a stereo VR view. Our multi-stage approach is inspired by state-of-the-art neural-network architectural design, like YOLO and Transformers. We demonstrate the generalization capabilities of SurgeoNet in challenging real-world scenarios, achieved solely through training on synthetic data. The approach can be easily extended to any new set of articulated surgical instruments. SurgeoNet's code and data are publicly available.
ISSN:2331-8422