Semi-Synchronous Speech and Pen Input - Watanabe, Y.; Iwata, K.; Nakagawa, R.; Shinoda, K.; Furui, S. (2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07, 2007)
This paper proposes a new interface method using semi-synchronous speech and pen input for mobile environments. In this interface, a user speaks while writing, where pen input complements speech to achieve higher recognition performance than speech alone. A multimodal recognition algorithm that can handle the asynchronicity of the two modes using a segment-based unification scheme is proposed. This method is evaluated under noisy conditions with four different pen-input interfaces: character, stroke, pen-touch, and point-to-character, each of which is assumed to be given for a phrase unit in speech. It is confirmed that the recognition accuracy is improved by the proposed method in comparison with that by speech alone in all the pen-input conditions.