Actions sketch: a novel action representation

• Published in: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) Vol. 1; pp. 984 - 989 vol. 1
• Main Authors: Alper Yilmaz, Mubarak Shah
• Format: Conference Proceeding
• Language: English
• Published: IEEE 2005
• Subjects: Computer science; Computer vision; Data mining; Hidden Markov models; Humans; Legged locomotion; Shape; Spatiotemporal phenomena; Surface treatment; Trajectory
• DOI: 10.1109/CVPR.2005.58
• ISBN: 0769523722; 9780769523729
• ISSN: 1063-6919

In this paper, we propose to model an action based on both the shape and the motion of the performing object. When the object performs an action in 3D, the points on the outer boundary of the object are projected as 2D (x, y) contour in the image plane. A sequence of such 2D contours with respect to time generates a spatiotemporal volume (STV) in (x, y, t), which can be treated as 3D object in the (x, y, t) space. We analyze STV by using the differential geometric surface properties to identify action descriptors capturing both spatial and temporal properties. A set of action descriptors is called an action sketch. The first step in our approach is to generate STV by solving the point correspondence problem between consecutive frames. The correspondences are determined using a two-step graph theoretical approach. After the STV is generated, actions descriptors are computed by analyzing the differential geometric properties of STV. Finally, using these descriptors, we perform action recognition, which is also formulated as graph theoretical problem. Several experimental results are presented to demonstrate our approach.