Multiple Human Association and Tracking From Egocentric and Complementary Top Views

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 44; no. 9; pp. 5225 - 5242
• Main Authors: Han, Ruize, Feng, Wei, Zhang, Yujun, Zhao, Jiewen, Wang, Song
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.09.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cameras; Collaboration; complementary view; Crowded scene surveillance; Distribution functions; egocentric perception; Graphical models; High altitude; horizontal view; human association; Integer programming; Mixed integer; Optimization; Spatial distribution; top view; Tracking; Trajectory; video surveillance; Videos; wearable cameras
• ISSN: 0162-8828; 1939-3539; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2021.3070562

Crowded scene surveillance can significantly benefit from combining egocentric-view and its complementary top-view cameras. A typical setting is an egocentric-view camera, e.g., a wearable camera on the ground capturing rich local details, and a top-view camera, e.g., a drone-mounted one from high altitude providing a global picture of the scene. To collaboratively analyze such complementary-view videos, an important task is to associate and track multiple people across views and over time, which is challenging and differs from classical human tracking, since we need to not only track multiple subjects in each video, but also identify the same subjects across the two complementary views. This paper formulates it as a constrained mixed integer programming problem, wherein a major challenge is how to effectively measure subjects similarity over time in each video and across two views. Although appearance and motion consistencies well apply to over-time association, they are not good at connecting two highly different complementary views. To this end, we present a spatial distribution based approach to reliable cross-view subject association. We also build a dataset to benchmark this new challenging task. Extensive experiments verify the effectiveness of our method.