Dense Trajectories and Motion Boundary Descriptors for Action Recognition

• Published in: International journal of computer vision Vol. 103; no. 1; pp. 60 - 79
• Main Authors: Wang, Heng, Kläser, Alexander, Schmid, Cordelia, Liu, Cheng-Lin
• Format: Journal Article
• Language: English
• Published: Boston Springer US 01.05.2013; Springer; Springer Nature B.V; Springer Verlag
• Subjects: Algorithms; Analysis; Applied sciences; Artificial Intelligence; Boundaries; Cameras; Classification; Computer Imaging; Computer Science; Computer science; control theory; systems; Computer Vision and Pattern Recognition; Datasets; Exact sciences and technology; Histograms; Image Processing and Computer Vision; Image processing systems; Pattern Recognition; Pattern Recognition and Graphics; Pattern recognition. Digital image processing. Computational geometry; Recognition; Representations; Sensors; Spacetime; State of the art; Studies; Trajectories; Vision; Vision systems; Action recognition; Motion boundary histograms; Dense trajectories; Multimedia; Computer vision; Action; Histogram; Motion estimation; Point shape; Video signal; Mobility; Sport; Behavioral analysis; Scene analysis; Classification; Object detection; Trajectory; Web site; Pattern extraction; Optical flow
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-012-0594-8

This paper introduces a video representation based on dense trajectories and motion boundary descriptors. Trajectories capture the local motion information of the video. A dense representation guarantees a good coverage of foreground motion as well as of the surrounding context. A state-of-the-art optical flow algorithm enables a robust and efficient extraction of dense trajectories. As descriptors we extract features aligned with the trajectories to characterize shape (point coordinates), appearance (histograms of oriented gradients) and motion (histograms of optical flow). Additionally, we introduce a descriptor based on motion boundary histograms (MBH) which rely on differential optical flow. The MBH descriptor shows to consistently outperform other state-of-the-art descriptors, in particular on real-world videos that contain a significant amount of camera motion. We evaluate our video representation in the context of action classification on nine datasets, namely KTH, YouTube, Hollywood2, UCF sports, IXMAS, UIUC, Olympic Sports, UCF50 and HMDB51. On all datasets our approach outperforms current state-of-the-art results.