Machine and deep learning for workflow recognition during surgery

• Published in: Minimally invasive therapy and allied technologies Vol. 28; no. 2; pp. 82 - 90
• Main Author: Padoy, Nicolas
• Format: Journal Article
• Language: English
• Published: England Taylor & Francis 04.03.2019
• Subjects: Activity recognition; clinician pose estimation; Computer Science; endoscopic video; Medical Imaging; operating room; RGB-D video; surgical control tower; surgical workflow; tool detection; RGB-D video; endoscopic video; Activity recognition; tool detection; surgical workflow; clinician pose estimation; surgical control tower; operating room
• ISSN: 1364-5706; 1365-2931; 1365-2931
• DOI: 10.1080/13645706.2019.1584116

Recent years have seen tremendous progress in artificial intelligence (AI), such as with the automatic and real-time recognition of objects and activities in videos in the field of computer vision. Due to its increasing digitalization, the operating room (OR) promises to directly benefit from this progress in the form of new assistance tools that can enhance the abilities and performance of surgical teams. Key for such tools is the recognition of the surgical workflow, because efficient assistance by an AI system requires this system to be aware of the surgical context, namely of all activities taking place inside the operating room. We present here how several recent techniques relying on machine and deep learning can be used to analyze the activities taking place during surgery, using videos captured from either endoscopic or ceiling-mounted cameras. We also present two potential clinical applications that we are developing at the University of Strasbourg with our clinical partners.