The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities

This paper describes a framework for modeling human activities as temporally structured processes. Our approach is motivated by the inherently hierarchical nature of human activities and the close correspondence between human actions and speech: We model action units using Hidden Markov Models, much...

Full description

Saved in:
Bibliographic Details
Published in2014 IEEE Conference on Computer Vision and Pattern Recognition pp. 780 - 787
Main Authors Kuehne, Hilde, Arslan, Ali, Serre, Thomas
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.06.2014
Subjects
Accuracy
Dairy products
Grammar
Hidden Markov models
Speech
Speech recognition
Sugar
Online AccessGet full text

Cover

More Information
Summary:This paper describes a framework for modeling human activities as temporally structured processes. Our approach is motivated by the inherently hierarchical nature of human activities and the close correspondence between human actions and speech: We model action units using Hidden Markov Models, much like words in speech. These action units then form the building blocks to model complex human activities as sentences using an action grammar. To evaluate our approach, we collected a large dataset of daily cooking activities: The dataset includes a total of 52 participants, each performing a total of 10 cooking activities in multiple real-life kitchens, resulting in over 77 hours of video footage. We evaluate the HTK toolkit, a state-of-the-art speech recognition engine, in combination with multiple video feature descriptors, for both the recognition of cooking activities (e.g., making pancakes) as well as the semantic parsing of videos into action units (e.g., cracking eggs). Our results demonstrate the benefits of structured temporal generative approaches over existing discriminative approaches in coping with the complexity of human daily life activities.
ISSN:1063-6919
DOI:10.1109/CVPR.2014.105