Patterns of Multimodal Input Usage in Non-Visual Information Navigation

• Published in: Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS'06) Vol. 6; p. 123c
• Main Authors: Xiaoyu Chen, Tremaine, M.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 2006
• Subjects: Error correction; Fingers; Graphical user interfaces; Guidelines; Haptic interfaces; Humans; Information systems; Navigation; Shape; Speech recognition
• ISBN: 0769525075; 9780769525075
• ISSN: 1530-1605; 1530-1605
• DOI: 10.1109/HICSS.2006.377

Multimodal input is known to be advantageous for graphical user interfaces, but its benefits for non-visual interaction are unknown. To explore this issue, an exploratory study was conducted with fourteen sighted subjects on a system that allows speech input and hand input on a touchpad. Findings include: (1) Users chose between these two input modalities based on the types of operations undertaken. Navigation operations were done primarily with touchpad input, while non-navigation instructions were carried out primarily using speech input. (2) Multimodal error correction was not prevalent. Repeating a failed operation until it succeeded and trying other methods in the same input modality were dominant error-correction strategies. (3) The modality learned first was not necessarily the primary modality used later, but a training order effect existed. These empirical results provide guidelines for designing non-visual multimodal input and create a comparison baseline for a subsequent study with blind users.