Less is More: Surgical Phase Recognition from Timestamp Supervision

• Published in: IEEE transactions on medical imaging Vol. 42; no. 6; p. 1
• Main Authors: Ding, Xinpeng, Yan, Xinjian, Wang, Zixun, Zhao, Wei, Zhuang, Jian, Xu, Xiaowei, Li, Xiaomeng
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.06.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Annotations; Costs; Frames; Iterative methods; Labels; Long short term memory; Manuals; Phases; Recognition; Supervision; Surgeons; Surgery; Surgery, Computer-Assisted; Surgical phase recognition; timestamp supervision; Training; Uncertainty; uncertainty estimation; Video; Videos
• ISSN: 0278-0062; 1558-254X
• DOI: 10.1109/TMI.2023.3242980

Surgical phase recognition is a fundamental task in computer-assisted surgery systems. Most existing works are under the supervision of expensive and time-consuming full annotations, which require the surgeons to repeat watching videos to find the precise start and end time for a surgical phase. In this paper, we introduce timestamp supervision for surgical phase recognition to train the models with timestamp annotations, where the surgeons are asked to identify only a single timestamp within the temporal boundary of a phase. This annotation can significantly reduce the manual annotation cost compared to the full annotations. To make full use of such timestamp supervisions, we propose a novel method called uncertainty-aware temporal diffusion (UATD) to generate trustworthy pseudo labels for training. Our proposed UATD is motivated by the property of surgical videos, i.e ., the phases are long events consisting of consecutive frames. To be specific, UATD diffuses the single labelled timestamp to its corresponding high confident ( i.e ., low uncertainty) neighbour frames in an iterative way. Our study uncovers unique insights of surgical phase recognition with timestamp supervision: 1) timestamp annotation can reduce 74% annotation time compared with the full annotation, and surgeons tend to annotate those timestamps near the middle of phases; 2) extensive experiments demonstrate that our method can achieve competitive results compared with full supervision methods, while reducing manual annotation costs; 3) less is more in surgical phase recognition, i.e ., less but discriminative pseudo labels outperform full but containing ambiguous frames; 4) the proposed UATD can be used as a plug-and-play method to clean ambiguous labels near boundaries between phases, and improve the performance of the current surgical phase recognition methods. Code and annotations obtained from surgeons are available at https://github.com/xmed-lab/ TimeStamp-Surgical.