Optical merger of direct vision with virtual images for scaled teleoperation

• Published in: IEEE transactions on visualization and computer graphics Vol. 12; no. 2; pp. 277 - 285
• Main Authors: Clanton, S.T., Wang, D.C., Chib, V.S., Matsuoka, Y., Stetten, G.D.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.03.2006; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; and virtual realities; Artificial; augmented; Biomedical imaging; Biomedical monitoring; Biomedical optical imaging; Computer Graphics; Corporate acquisitions; Devices; Equipment Design; Equipment Failure Analysis; Feedback; Humans; image display; Image Enhancement - instrumentation; Image Interpretation, Computer-Assisted - instrumentation; Imaging, Three-Dimensional - instrumentation; Medical equipment; medical information systems; Medical robotics; Numerical Analysis, Computer-Assisted; Optical feedback; Patient monitoring; Patients; real time; Robot kinematics; Robotics - instrumentation; Robotics - methods; Signal Processing, Computer-Assisted; Software; Sonics; Subtraction Technique; Telemedicine - instrumentation; Telemedicine - methods; Ultrasonic imaging; Ultrasound; User-Computer Interface; Visual; Visual Perception - physiology; Visualization
• ISSN: 1077-2626; 1941-0506
• DOI: 10.1109/TVCG.2006.35

Scaled teleoperation is increasingly prevalent in medicine, as well as in other applications of robotics. Visual feedback in such systems is essential and should make maximal use of natural hand-eye coordination. This paper describes a new method of visual feedback for scaled teleoperation in which the operator manipulates the handle of a remote tool in the presence of a registered virtual image of the target in real time. The method adapts a concept already used successfully in a new medical device called the sonic flashlight, which permits direct in situ visualization of ultrasound during invasive procedures. The sonic flashlight uses a flat-panel monitor and a half-silvered mirror to merge the visual outer surface of a patient with a simultaneous ultrasound scan of the patient's interior. Adapting the concept to scaled teleoperation involves removing the imaging device and the target to a remote location and adding a master-slave control device. This permits the operator to see his hands, along with what appears to be the tool, and the target, merged in a workspace that preserves natural hand-eye coordination. Three functioning prototypes are described, one based on ultrasound and two on light microscopy. The limitations and potential of the new approach are discussed.