Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 33; no. 11; pp. 2287 - 2301
• Main Authors: Morris, B. T., Trivedi, M. M.
• Format: Journal Article
• Language: English
• Published: Los Alamitos, CA IEEE 01.11.2011; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: abnormality detection; activity prediction; Applied sciences; Artificial intelligence; Computer science; control theory; systems; Construction; Exact sciences and technology; Hidden Markov models; Intelligence; Learning; Mathematical models; Pattern analysis; Pattern recognition. Digital image processing. Computational geometry; Probabilistic logic; real-time activity analysis; Regression; Retraining; Sparse matrices; Studies; Surveillance; Training; Trajectories; Trajectory; Trajectory clustering; trajectory learning; Recurrence; Vocabulary; Image processing; activity prediction; Modeling; Adaptive method; trajectory learning; Incomplete information; Localization; Time analysis; Monitoring; Mixed distribution; Cluster; Video cameras; Real time; Long term; Missing data; real-time activity analysis; Image analysis; Gaussian process; Surveillance; Trajectory clustering; abnormality detection; Scene analysis; Hidden Markov model; Maximum likelihood
• ISSN: 0162-8828; 1939-3539; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2011.64

Society is rapidly accepting the use of video cameras in many new and varied locations, but effective methods to utilize and manage the massive resulting amounts of visual data are only slowly developing. This paper presents a framework for live video analysis in which the behaviors of surveillance subjects are described using a vocabulary learned from recurrent motion patterns, for real-time characterization and prediction of future activities, as well as the detection of abnormalities. The repetitive nature of object trajectories is utilized to automatically build activity models in a 3-stage hierarchical learning process. Interesting nodes are learned through Gaussian mixture modeling, connecting routes formed through trajectory clustering, and spatio-temporal dynamics of activities probabilistically encoded using hidden Markov models. Activity models are adapted to small temporal variations in an online fashion using maximum likelihood regression and new behaviors are discovered from a periodic retraining for long-term monitoring. Extensive evaluation on various data sets, typically missing from other work, demonstrates the efficacy and generality of the proposed framework for surveillance-based activity analysis.