Real-time markerless tracking for augmented reality: the virtual visual servoing framework

• Published in: IEEE transactions on visualization and computer graphics Vol. 12; no. 4; pp. 615 - 628
• Main Authors: Comport, A.I., Marchand, E., Pressigout, M., Chaumette, F.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.07.2006; The Institute of Electrical and Electronics Engineers, Inc. (IEEE); Institute of Electrical and Electronics Engineers
• Subjects: Algorithms; Augmented reality; Cameras; Computer Graphics; Computer Science; Computer Systems; Computer Vision and Pattern Recognition; Costs; Delay; Feedback; Image Enhancement - methods; Image Interpretation, Computer-Assisted - methods; Image Processing; Imaging, Three-Dimensional - methods; Information Storage and Retrieval - methods; Layout; Machine vision; Mathematical models; model-based tracking; model-free tracking; Real time; Real time systems; Robotics; Robust control; robust estimators; Robustness; Signal Processing, Computer-Assisted; Studies; Three dimensional; Tracking; User-Computer Interface; virtual visual servoing; Vision systems; Visual; Visual servoing; model-based tracking; model-free tracking; real-time; Augmented reality; robust estimators; virtual visual servoing
• ISSN: 1077-2626; 1941-0506
• DOI: 10.1109/TVCG.2006.78

Tracking is a very important research subject in a real-time augmented reality context. The main requirements for trackers are high accuracy and little latency at a reasonable cost. In order to address these issues, a real-time, robust, and efficient 3D model-based tracking algorithm is proposed for a "video see through" monocular vision system. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose. In this paper, nonlinear pose estimation is formulated by means of a virtual visual servoing approach. In this context, the derivation of point-to-curves interaction matrices are given for different 3D geometrical primitives including straight lines, circles, cylinders, and spheres. A local moving edges tracker is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating an M-estimator into the visual control law via an iteratively reweighted least squares implementation. This approach is then extended to address the 3D model-free augmented reality problem. The method presented in this paper has been validated on several complex image sequences including outdoor environments. Results show the method to be robust to occlusion, changes in illumination, and mistracking.