Hierarchical Bayesian LSTM for Head Trajectory Prediction on Omnidirectional Images

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 44; no. 11; pp. 7563 - 7580
• Main Authors: Yang, Li, Xu, Mai, Guo, Yichen, Deng, Xin, Gao, Fangyuan, Guan, Zhenyu
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.11.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Bayes methods; Bayesian analysis; Computational modeling; Datasets; Head; Head movement; head trajectory; Hidden Markov models; hierarchical Bayesian inference; Magnetic heads; Modelling; Normal distribution; Omnidirectional images; Predictive models; Statistical inference; Trajectory
• DOI: 10.1109/TPAMI.2021.3117019
• ISSN: 0162-8828; 1939-3539

When viewing omnidirectional images (ODIs), viewers can access different viewports via head movement (HM), which sequentially forms head trajectories in spatial-temporal domain. Thus, head trajectories play a key role in modeling human attention on ODIs. In this paper, we establish a large-scale dataset collecting 21,600 head trajectories on 1,080 ODIs. By mining our dataset, we find two important factors influencing head trajectories, i.e., temporal dependency and subject-specific variance. Accordingly, we propose a novel approach integrating hierarchical Bayesian inference into long short-term memory (LSTM) network for head trajectory prediction on ODIs, which is called HiBayes-LSTM. In HiBayes-LSTM, we develop a mechanism of Future Intention Estimation (FIE), which captures the temporal correlations from previous, current and estimated future information, for predicting viewport transition. Additionally, a training scheme called Hierarchical Bayesian inference (HBI) is developed for modeling inter-subject uncertainty in HiBayes-LSTM. For HBI, we introduce a joint Gaussian distribution in a hierarchy, to approximate the posterior distribution over network weights. By sampling subject-specific weights from the approximated posterior distribution, our HiBayes-LSTM approach can yield diverse viewport transition among different subjects and obtain multiple head trajectories. Extensive experiments validate that our HiBayes-LSTM approach significantly outperforms 9 state-of-the-art approaches for trajectory prediction on ODIs, and then it is successfully applied to predict saliency on ODIs.