Egocentric Human Trajectory Forecasting With a Wearable Camera and Multi-Modal Fusion

In this letter, we address the problem of forecasting the trajectory of an egocentric camera wearer (ego-person) in crowded spaces. The trajectory forecasting ability learned from the data of different camera wearers walking around in the real world can be transferred to assist visually impaired peo...

Full description

Saved in:
Bibliographic Details
Published inIEEE robotics and automation letters Vol. 7; no. 4; pp. 8799 - 8806
Main Authors Qiu, Jianing, Chen, Lipeng, Gu, Xiao, Lo, Frank P.-W., Tsai, Ya-Yen, Sun, Jiankai, Liu, Jiaqi, Lo, Benny
Format Journal Article
LanguageEnglish
Published Piscataway IEEE 01.10.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects
Cameras
Coders
egocentric vision
Encoders-Decoders
Forecasting
Human trajectory forecasting
Mathematical models
multi-modal learning
Navigation
Neural networks
People with disabilities
Robot vision systems
Robots
Semantics
Trajectory
Transformers
Online AccessGet full text

Cover

More Information
Summary:In this letter, we address the problem of forecasting the trajectory of an egocentric camera wearer (ego-person) in crowded spaces. The trajectory forecasting ability learned from the data of different camera wearers walking around in the real world can be transferred to assist visually impaired people in navigation, as well as to instill human navigation behaviours in mobile robots, enabling better human-robot interactions. To this end, a novel egocentric human trajectory forecasting dataset was constructed, containing real trajectories of people navigating in crowded spaces wearing a camera, as well as extracted rich contextual data. We extract and utilize three different modalities to forecast the trajectory of the camera wearer, i.e., his/her past trajectory, the past trajectories of nearby people, and the environment such as the scene semantics or the depth of the scene. A Transformer-based encoder-decoder neural network model, integrated with a novel cascaded cross-attention mechanism that fuses multiple modalities, has been designed to predict the future trajectory of the camera wearer. Extensive experiments have been conducted, with results showing that our model outperforms the state-of-the-art methods in egocentric human trajectory forecasting.
ISSN:2377-3766
2377-3766
DOI:10.1109/LRA.2022.3188101