Scene Modeling and Augmented Virtuality Interface for Telerobotic Satellite Servicing

• Published in: IEEE robotics and automation letters Vol. 3; no. 4; pp. 4241 - 4248
• Main Authors: Vagvolgyi, Balazs P., Pryor, Will, Reedy, Ryan, Niu, Wenlong, Deguet, Anton, Whitcomb, Louis L., Leonard, Simon, Kazanzides, Peter
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.10.2018; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cameras; Delays; Extreme environments; Fixtures; Modelling; Robot vision systems; Satellites; Solid modeling; space robotics and automation; Telemetry; Telerobotics; Telerobotics and teleoperation; Three dimensional models; Virtual reality; virtual reality and interfaces; Visual perception; Visualization
• ISSN: 2377-3766; 2377-3766
• DOI: 10.1109/LRA.2018.2864358

Teleoperation in extreme environments can be hindered by limitations in telemetry and in operator perception of the remote environment. Often, the primary mode of perception is visual feedback from remote cameras, which do not always provide suitable views and are subject to telemetry delays. To address these challenges, we propose to build a model of the remote environment and provide an augmented virtuality visualization system that augments the model with projections of real camera images. The approach is demonstrated in a satellite servicing scenario, with a multisecond round-trip telemetry delay between the operator on Earth and the satellite on orbit. The scene modeling enables both virtual fixtures to assist the human operator and augmented virtuality visualization that allows the operator to teleoperate a virtual robot from a convenient virtual viewpoint, with the delayed camera images projected onto the three-dimensional model. Experiments on a ground-based telerobotic platform, with software-created telemetry delays, indicate that the proposed method leads to better teleoperation performance with 30% better blade alignment and 50% reduction in task execution time compared to the baseline case where visualization is restricted to the available camera views.