Reaching Across the Communication Gap: Evaluating How Augmented Reality Shared Gestural Spaces Impact Gesture and Language Usage

• Published in: Proceedings - Frontiers in Education Conference pp. 1 - 6
• Main Authors: Hordemann, Glen, Quek, Francis, Powell, Larry
• Format: Conference Proceeding
• Language: English
• Published: IEEE 18.10.2023
• Subjects: augmented reality; Computer aided instruction; COVID-19; Human computer interaction (HCI); Human-centered computing; Indexes; Interaction paradigms; Maintenance engineering; Telepresence; Visualization; Web conferencing
• DOI: 10.1109/FIE58773.2023.10343016

Instructional discourse around physically-based material typically involves the use of gestures to focus attention in the joint visual space of the instructor and the student. Such gestures are performed in conjunction with speech, thus situating the discourse in the space of what the student is learning or doing. This kind of situated discourse is missing in distance learning. The lack of situatedness between conversants contributes to communication inefficiencies, as was recently highlighted when classes began using Zoom during COVID. For instruction that involves hands-on learning, the inability for instructors to index the student's space and vice versa is a more significant problem. The use of telepresence technologies allows this spatial gap to be bridged but can often leave a communication gap. Little evaluation has been made into how these mechanisms alter gesture and speech. This paper explores how giving the instructor situated gestural access to the student's workspace in distant learning settings alters gesture and speech, with the goal of guiding design of future systems that are mindful of how these systems work, not simply whether they work. Our research is based on a system that supports situated instructional discourse through augmented reality gesturing. The instructor is able to see the student's workspace and perform gestures that are superposed into the student's workspace. This shared gestural space is used in tandem with a traditional video chat. Two spatially-focused tasks were examined: a two-dimension problem that required varying levels of assistance, and a three-dimension problem with complete guidance by the instructor needed. We evaluated how gestures were used to augment verbal communication and the types of gestures and verbal utterances used. We found that static pointing G-hand deictic gestures were more common in the augmented reality mode than in the co-situated mode and complex spatial structuring gestures (e.g. orientation gestures) occurred more often in the co-situated mode than the augmented reality mode. Additionally, students provided a range of body postures and working orientations to encourage or discourage invasion of their personal workspace in co-situated settings, and several subjects indicated that the augmented reality made them feel more comfortable with sharing their gestural space. These results provide a framework to guide the design of future improvements of telepresence systems. Subjective statements of comfort by subjects also suggest this may be a profitable avenue for exploration with students who are reluctant to engage physically with instructors.