A task-driven intelligent workspace system to provide guidance feedback

• Published in: Computer vision and image understanding Vol. 114; no. 5; pp. 520 - 534
• Main Authors: Ryoo, M.S., Grauman, Kristen, Aggarwal, J.K.
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.05.2010
• Subjects: Activity recognition; Assembling; Assembly; C (programming language); Event analysis; Feedback; Feedback generation; Guidance feedback messages; Intelligent environments; Intelligent workspaces; Messages; Tasks; Tightening; Activity recognition; Intelligent environments; Intelligent workspaces; Feedback generation; Guidance feedback messages; Event analysis
• ISSN: 1077-3142; 1090-235X
• DOI: 10.1016/j.cviu.2009.12.009

We introduce a vision-based human–computer interaction system which collaborates with a user by providing feedback during user activities. We design an intelligent workspace system that analyzes the context of a user’s tasks and generates appropriate feedback guiding the user to complete the tasks correctly. While a user is performing the high-level activity, our system analyzes what sub-events the user has already completed and what sub-events are needed next in order for the user to finish the activity. Based on this analysis, the system generates hierarchical feedback messages to assist the user. We test our system on three different types of tasks: tightening nuts on a wheel of a car, assembling a toy spaceship with LEGO blocks, and assembling a laptop computer with its parts. Experimental results demonstrate the accuracy of the vision-based activity analysis, and a comparative user study shows that our system’s feedback enables users to complete assembly tasks with significantly improved efficiency.