Combining multivariate statistics and the think-aloud protocol to assess Human-Computer Interaction barriers in symptom checkers

• Published in: Journal of biomedical informatics Vol. 74; pp. 104 - 122
• Main Authors: Marco-Ruiz, Luis, Bønes, Erlend, de la Asunción, Estela, Gabarron, Elia, Aviles-Solis, Juan Carlos, Lee, Eunji, Traver, Vicente, Sato, Keiichi, Bellika, Johan G.
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.10.2017; Elsevier
• Subjects: Clinical Decision Support Systems; Clinical medical disciplines: 750; Decision Support Systems, Clinical; Human Computer Interaction; Humans; Klinisk medisinske fag: 750; Man-Machine Systems; Medical disciplines: 700; Medisinske Fag: 700; Multivariate Analysis; Principal Component Analysis; Symptom checkers; Usability testing; VDP; Usability testing; Symptom checkers; Clinical Decision Support Systems; Human Computer Interaction
• ISSN: 1532-0464; 1532-0480; 1532-0480
• DOI: 10.1016/j.jbi.2017.09.002

[Display omitted] •A cost-effective methodology to test symptom checkers’ interfaces is presented.•Multivariate statistics were used to detect areas influencing technology acceptance.•Think-aloud was executed to analyze these areas for understanding usability problems.•The method allows for optimizing the use of resources in complex usability studies. Symptom checkers are software tools that allow users to submit a set of symptoms and receive advice related to them in the form of a diagnosis list, health information or triage. The heterogeneity of their potential users and the number of different components in their user interfaces can make testing with end-users unaffordable. We designed and executed a two-phase method to test the respiratory diseases module of the symptom checker Erdusyk. Phase I consisted of an online test with a large sample of users (n=53). In Phase I, users evaluated the system remotely and completed a questionnaire based on the Technology Acceptance Model. Principal Component Analysis was used to correlate each section of the interface with the questionnaire responses, thus identifying which areas of the user interface presented significant contributions to the technology acceptance. In the second phase, the think-aloud procedure was executed with a small number of samples (n=15), focusing on the areas with significant contributions to analyze the reasons for such contributions. Our method was used effectively to optimize the testing of symptom checker user interfaces. The method allowed kept the cost of testing at reasonable levels by restricting the use of the think-aloud procedure while still assuring a high amount of coverage. The main barriers detected in Erdusyk were related to problems understanding time repetition patterns, the selection of levels in scales to record intensities, navigation, the quantification of some symptom attributes, and the characteristics of the symptoms.