Display-camera calibration using eye reflections and geometry constraints

• Published in: Computer vision and image understanding Vol. 115; no. 6; pp. 835 - 853
• Main Authors: Nitschke, Christian, Nakazawa, Atsushi, Takemura, Haruo
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier Inc 01.06.2011; Elsevier
• Subjects: Applied sciences; Artificial intelligence; Calibration; Catadioptric imaging; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Corneal reflection; Devices; Display-camera system; Exact sciences and technology; Eye pose estimation; Eyes; Human eye model; Image analysis; Mathematical models; Pattern recognition. Digital image processing. Computational geometry; Reconstruction; Reflection; Software; Strategy; Visual system; Visual system; Catadioptric imaging; Image analysis; Human eye model; Eye pose estimation; Calibration; Display-camera system; Corneal reflection; Gaze; Man machine system; Image processing; Modeling; Image reconstruction; Posture; User interface; Imaging; Facies; Object oriented; Illumination; Retroreflector; Face recognition; Linear programming; Video cameras; Luminance; Mirror; Ambiguity
• ISSN: 1077-3142; 1090-235X
• DOI: 10.1016/j.cviu.2011.02.008

► Novel idea to analyze corneal reflections of screen illumination in the human eye. ► Novel method for calibrating the geometric relation in display-camera setups. ► Method does not require any user interaction or awareness and special hardware. ► Optimization framework leads to mutual benefit for eye and display pose estimation. ► Findings might be applied to other works in eye feature and reflection analysis. In this paper, we describe a novel method for calibrating display-camera setups from reflections in a user’s eyes. Combining both devices creates a capable controlled illumination system that enables a range of interesting vision applications in non-professional environments, including object/face reconstruction and human–computer interaction. One major issue barring such systems from average homes is the geometric calibration to obtain the pose of the display which requires special hardware and tedious user interaction. Our proposed approach eliminates this requirement by introducing the novel idea of analyzing screen reflections in the cornea of the human eye, a mirroring device that is always available. We employ a simple shape model to recover pose and reflection characteristics of the eye. Thorough experimental evaluation shows that the basic strategy results in a large error and discusses possible reasons. Based on the findings, a non-linear optimization strategy is developed that exploits geometry constraints within the system to considerably improve the initial estimate. It further allows to automatically resolve an inherent ambiguity that arises in image-based eye pose estimation. The strategy may also be integrated to improve spherical mirror calibration. We describe several comprehensive experimental studies which show that the proposed method performs stably with respect to varying subjects, display poses, eye positions, and gaze directions. The results are feasible and should be sufficient for many applications. In addition, the findings provide general insight on the application of eye reflections for geometric reconstruction.