Design of human-centric adaptive multimodal interfaces

• Published in: International journal of human-computer studies Vol. 69; no. 12; pp. 854 - 869
• Main Authors: Kong, J., Zhang, W.Y., Yu, N., Xia, X.J.
• Format: Journal Article
• Language: English
• Published: Oxford Elsevier Ltd 01.12.2011; Elsevier
• Subjects: Adaptive multimodal interfaces; Adaptive user interface design; Applied sciences; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Exact sciences and technology; Human–computer interaction; Information systems. Data bases; Linear programming; Memory organisation. Data processing; Mobile users; Optimization; Software; User study; Utility evaluation; Adaptive user interface design; Adaptive multimodal interfaces; User study; Mobile users; Linear programming; Utility evaluation; Optimization; Human–computer interaction; Input output; Mode coupling; Ontology; Man machine system; Expert system; Modeling; Adaptive method; User interface; Multimodal interface; User need; Transmission protocol; Program specification; Optimal solution; Heuristic method; Data fusion; Human-computer interaction; Service quality; User behavior; Mobile computing
• ISSN: 1071-5819; 1095-9300
• DOI: 10.1016/j.ijhcs.2011.07.006

Multimodal interfaces have attracted more and more attention. Most researches focus on each interaction mode independently and then fuse information at the application level. Recently, several frameworks and models have been proposed to support the design and development of multimodal interfaces. However, it is challenging to provide automatic modality adaptation in multimodal interfaces. Existing approaches are using rule-based specifications to define the adaptation of input/output modalities. Rule-based specifications have the problems of completeness and coherence. Distinct from previous work, this paper presents a novel approach that quantifies the user preference of each modality and considers the adaptation as an optimization issue that searches for a set of input/output modalities matching user's preference. Our approach applies a cross-layer design, which considers the adaptation from the perspectives of the interaction context, available system resources, and QoS requirements. Furthermore, our approach supports human-centric adaptation. A user can report the preference of a modality so that selected modalities fit user's personal needs. An optimal solution and a heuristic algorithm have been developed to automatically select an appropriate set of modality combinations under a specific situation. We have designed a framework based on the heuristic algorithm and existing ontology, and applied the framework to conduct a utility evaluation, in which we have employed a within-subject experiment. Fifty participants were invited to go through three scenarios and compare automatically selected modalities with randomly selected modalities. The results from the experiment show that users perceived the automatically selected modalities as appropriate and satisfactory. ► A cross-layer design includes application, system and network layers. ► Human-centric adaptation supports customizable modality selection that fits personal needs. ► A utility evaluation proves the promising performance of the proposed approach.