Enabling Product Recognition and Tracking Based on Text Detection for Mobile Augmented Reality

• Published in: IEEE access Vol. 10; pp. 98769 - 98782
• Main Authors: Hwang, Sangwon, Lee, Jisun, Kang, Seungwoo
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Augmented reality; Character recognition; Feature extraction; Image recognition; Mobile augmented reality; Optical flow (image analysis); Packages; Product delivery; Product names; product recognition; product tracking; Real-time systems; Recognition; Target recognition; Target tracking; text detection; Text recognition; Tracking
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2022.3205344

We propose a system that supports real-time product recognition and tracking based on text detection for mobile augmented reality. To accurately distinguish products with visually similar packages, we develop a method that recognizes product names by utilizing the characteristics of texts printed on the product packages. It first filters out irrelevant products and effectively ranks candidate products through an inverted index search. We significantly reduce processing overhead by selectively performing product name recognition. In addition, we present an optical-flow-based method that enables efficient and responsive product tracking. Our evaluation shows that the proposed system achieves significantly better product recognition accuracy (80%) compared to alternative solutions, Vuforia (55.4%) and MobileNetV2 (69.6%). We also show that it achieves reasonable tracking accuracy and processing latency to support quality mobile AR experiences.