Performance Envelopes of Virtual Keyboard Text Input Strategies in Virtual Reality

• Published in: 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) pp. 289 - 300
• Main Authors: Dudley, John, Benko, Hrvoje, Wigdor, Daniel, Kristensson, Per Ola
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.10.2019
• Subjects: Augmented reality; Decoding; head mounted display; Indexes; Keyboards; Layout; Performance evaluation; text entry; Tracking; virtual reality
• DOI: 10.1109/ISMAR.2019.00027

Virtual and Augmented Reality deliver engaging interaction experiences that can transport and extend the capabilities of the user. To ensure these paradigms are more broadly usable and effective, however, it is necessary to also deliver many of the conventional functions of a smartphone or personal computer. It remains unclear how conventional input tasks, such as text entry, can best be translated into virtual and augmented reality. In this paper we examine the performance potential of four alternative text entry strategies in virtual reality (VR). These four strategies are selected to provide full coverage of two fundamental design dimensions: i) physical surface association; and ii) number of engaged fingers. Specifically, we examine typing with index fingers on a surface and in mid-air and typing using all ten fingers on a surface and in mid-air. The central objective is to evaluate the human performance potential of these four typing strategies without being constrained by current tracking and statistical text decoding limitations. To this end we introduce an auto-correction simulator that uses knowledge of the stimulus to emulate statistical text decoding within constrained experimental parameters and use high-precision motion tracking hardware to visualise and detect fingertip interactions. We find that alignment of the virtual keyboard with a physical surface delivers significantly faster entry rates over a mid-air keyboard. Also, users overwhelmingly fail to effectively engage all ten fingers in mid-air typing, resulting in slower entry rates and higher error rates compared to just using two index fingers. In addition to identifying the envelopes of human performance for the four strategies investigated, we also provide a detailed analysis of the underlying features that distinguish each strategy in terms of its performance and behaviour.