Affine transformation of virtual 3D object using 2D localization of fingertips

• Published in: Virtual Reality & Intelligent Hardware Vol. 2; no. 6; pp. 534 - 555
• Main Authors: Mohammad Mahmudul Alam, S.M. Mahbubur Rahman
• Format: Journal Article
• Language: English
• Published: KeAi Communications Co., Ltd 01.12.2020
• Subjects: Affine transformation; Detection of fingertips; Detection of hand; Human-computer interaction; Virtual reality
• ISSN: 2096-5796
• DOI: 10.1016/j.vrih.2020.10.001

Background: Interactions with virtual 3D objects in the virtual reality (VR) environment using the gesture of fingers captured in a wearable 2D camera have emerging applications in real-life. Method: This paper presents an approach of a two-stage convolutional neural network, one for the detection of hand and another for the fingertips. One purpose of VR environments is to transform a virtual 3D object with affine parameters by using the gesture of thumb and index fingers. Results: To evaluate the performance of the proposed system, one existing, and another developed egocentric fingertip databases are employed so that learning involves large variations that are common in real-life. Experimental results show that the proposed fingertip detection system outperforms the existing systems in terms of the precision of detection. Conclusion: The interaction performance of the proposed system in the VR environment is higher than that of the existing systems in terms of estimation error and correlation between the ground truth and estimated affine parameters.