SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research - Extended Abstract

We introduce an open-source system called Sigma (short for "Situ-ated Interactive Guidance, Monitoring, and Assistance") as a plat-form for conducting research on task-assistive agents in mixed-reality scenarios. The system leverages the sensing and rendering affordances of a head-mounted...

Full description

Saved in:
Bibliographic Details
Published in2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) pp. 889 - 890
Main Authors Bohus, Dan, Andrist, Sean, Saw, Nick, Paradiso, Ann, Chakraborty, Ishani, Rad, Mahdi
Format Conference Proceeding
LanguageEnglish
Published IEEE 16.03.2024
Subjects
artificial intelligence
Interactive systems
Mixed reality
Multimodal sensors
Solid modeling
task assistance
Three-dimensional displays
User interfaces
Virtual reality
Online AccessGet full text

Cover

More Information
Summary:We introduce an open-source system called Sigma (short for "Situ-ated Interactive Guidance, Monitoring, and Assistance") as a plat-form for conducting research on task-assistive agents in mixed-reality scenarios. The system leverages the sensing and rendering affordances of a head-mounted mixed reality device in conjunction with large language and vision models to guide users step by step through procedural tasks. By open-sourcing the system, we aim to lower the barrier to entry, accelerate research in this space, and chart a path towards community-driven end-to-end evaluation of large language, vision, and multimodal models in the context of real-world interactive applications.
DOI:10.1109/VRW62533.2024.00241