ViFiT: Reconstructing Vision Trajectories from IMU and Wi-Fi Fine Time Measurements

• Published in: arXiv.org
• Main Authors: Cao, Bryan Bo, Alali, Abrar, Liu, Hansi, Meegan, Nicholas, Gruteser, Marco, Dana, Kristin, Ashwin Ashok, Jain, Shubham
• Format: Paper Journal Article
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 04.10.2023
• Subjects: Bandwidths; Boxes; Cameras; Coders; Computer Science - Computer Vision and Pattern Recognition; Computer Science - Multimedia; Computer vision; Datasets; Encoders-Decoders; Frames (data processing); Image reconstruction; Reduction; Time measurement; Tracking; Traffic surveillance; Transformers
• ISSN: 2331-8422
• DOI: 10.48550/arxiv.2310.03140

Tracking subjects in videos is one of the most widely used functions in camera-based IoT applications such as security surveillance, smart city traffic safety enhancement, vehicle to pedestrian communication and so on. In the computer vision domain, tracking is usually achieved by first detecting subjects with bounding boxes, then associating detected bounding boxes across video frames. For many IoT systems, images captured by cameras are usually sent over the network to be processed at a different site that has more powerful computing resources than edge devices. However, sending entire frames through the network causes significant bandwidth consumption that may exceed the system bandwidth constraints. To tackle this problem, we propose ViFiT, a transformer-based model that reconstructs vision bounding box trajectories from phone data (IMU and Fine Time Measurements). It leverages a transformer ability of better modeling long-term time series data. ViFiT is evaluated on Vi-Fi Dataset, a large-scale multimodal dataset in 5 diverse real world scenes, including indoor and outdoor environments. To fill the gap of proper metrics of jointly capturing the system characteristics of both tracking quality and video bandwidth reduction, we propose a novel evaluation framework dubbed Minimum Required Frames (MRF) and Minimum Required Frames Ratio (MRFR). ViFiT achieves an MRFR of 0.65 that outperforms the state-of-the-art approach for cross-modal reconstruction in LSTM Encoder-Decoder architecture X-Translator of 0.98, resulting in a high frame reduction rate as 97.76%.