Marker-less real-time intra-operative camera and hand-eye calibration procedure for surgical augmented reality

• Published in: Healthcare technology letters Vol. 6; no. 6; pp. 255 - 260
• Main Authors: Kalia, Megha, Mathur, Prateek, Navab, Nassir, Salcudean, Septimiu E
• Format: Journal Article
• Language: English
• Published: England The Institution of Engineering and Technology 01.12.2019; John Wiley & Sons, Inc; Wiley
• Subjects: Augmented reality; augmented reality rendering; average target registration error; biomedical optical imaging; Calibration; camera calibration; camera intrinsic matrix estimation; Cameras; da Vinci robot; Design; endoscope; endoscopes; Hair; hand-eye calibration procedure; hand-eye transformation; high visual error; image registration; marker-less real-time intra-operative camera; medical image processing; medical robotics; Methods; phantoms; pre-operative medical data; Prostate; prostate phantom; rendering (computer graphics); robot vision; Special Issue: Papers from the 13th Workshop on Augmented Environments for Computer Assisted Interventions; subsequent gradient descent steps; Surgeons; surgery; surgical augmented reality; virtual rendered tool tip; image registration; medical robotics; average target registration error; augmented reality; cameras; phantoms; hand-eye transformation; endoscopes; calibration; rendering (computer graphics); pre-operative medical data; medical image processing; virtual rendered tool tip; augmented reality rendering; camera calibration; subsequent gradient descent steps; robot vision; marker-less real-time intra-operative camera; prostate phantom; surgical augmented reality; da Vinci robot; camera intrinsic matrix estimation; high visual error; hand-eye calibration procedure; endoscope; surgery; biomedical optical imaging
• DOI: 10.1049/htl.2019.0094
• ISSN: 2053-3713

Accurate medical Augmented Reality (AR) rendering requires two calibrations, a camera intrinsic matrix estimation and a hand-eye transformation. We present a unified, practical, marker-less, real-time system to estimate both these transformations during surgery. For camera calibration we perform calibrations at multiple distances from the endoscope, pre-operatively, to parametrize the camera intrinsic matrix as a function of distance from the endoscope. Then, we retrieve the camera parameters intra-operatively by estimating the distance of the surgical site from the endoscope in less than 1 s. Unlike in prior work, our method does not require the endoscope to be taken out of the patient; for the hand-eye calibration, as opposed to conventional methods that require the identification of a marker, we make use of a rendered tool-tip in 3D. As the surgeon moves the instrument and observes the offset between the actual and the rendered tool-tip, they can select points of high visual error and manually bring the instrument tip to match the virtual rendered tool tip. To evaluate the hand-eye calibration, 5 subjects carried out the hand-eye calibration procedure on a da Vinci robot. Average Target Registration Error of approximately 7mm was achieved with just three data points.