The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool
[Display omitted] •Comparison of Concurrent and Retrospective Think Aloud in usability testing of HIT.•Concurrent and Retrospective Think Aloud differ in usability problem detection scope.•Concurrent Think Aloud has higher effectiveness in terms of system redesign.•Retrospective Think Aloud is promo...
Saved in:
Published in | Journal of biomedical informatics Vol. 55; pp. 1 - 10 |
---|---|
Main Authors | , , |
Format | Journal Article |
Language | English |
Published |
United States
Elsevier Inc
01.06.2015
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | [Display omitted]
•Comparison of Concurrent and Retrospective Think Aloud in usability testing of HIT.•Concurrent and Retrospective Think Aloud differ in usability problem detection scope.•Concurrent Think Aloud has higher effectiveness in terms of system redesign.•Retrospective Think Aloud is promoted for research on expert user customized design.
To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes.
Sixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method.
The usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool’s screens. RTA detected unique issues concerning system match with subjects’ language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians’ query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p<0.05) respectively, with an average difference of 42% less time spent per subject compared to RTA. In addition, CTA was more thorough in detecting usability problems of a moderate (0.85 vs. 0.7) and severe nature (0.71 vs. 0.57).
In this study, the CTA is more effective in usability-problem detection and provided clarification of intensive care physician query design strategies to inform redesign of the query tool. However, CTA does not outperform RTA. The RTA additionally elucidated unique usability problems and new user requirements. Based on the results of this study, we recommend the use of CTA in formative usability evaluation studies of health information technology. However, we recommend further research on the application of RTA in usability studies with regard to user expertise and experience when focusing on user profile customized (re)design. |
---|---|
AbstractList | [Display omitted]
•Comparison of Concurrent and Retrospective Think Aloud in usability testing of HIT.•Concurrent and Retrospective Think Aloud differ in usability problem detection scope.•Concurrent Think Aloud has higher effectiveness in terms of system redesign.•Retrospective Think Aloud is promoted for research on expert user customized design.
To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes.
Sixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method.
The usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool’s screens. RTA detected unique issues concerning system match with subjects’ language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians’ query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p<0.05) respectively, with an average difference of 42% less time spent per subject compared to RTA. In addition, CTA was more thorough in detecting usability problems of a moderate (0.85 vs. 0.7) and severe nature (0.71 vs. 0.57).
In this study, the CTA is more effective in usability-problem detection and provided clarification of intensive care physician query design strategies to inform redesign of the query tool. However, CTA does not outperform RTA. The RTA additionally elucidated unique usability problems and new user requirements. Based on the results of this study, we recommend the use of CTA in formative usability evaluation studies of health information technology. However, we recommend further research on the application of RTA in usability studies with regard to user expertise and experience when focusing on user profile customized (re)design. To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes. Sixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method. The usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool's screens. RTA detected unique issues concerning system match with subjects' language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians' query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p<0.05) respectively, with an average difference of 42% less time spent per subject compared to RTA. In addition, CTA was more thorough in detecting usability problems of a moderate (0.85 vs. 0.7) and severe nature (0.71 vs. 0.57). In this study, the CTA is more effective in usability-problem detection and provided clarification of intensive care physician query design strategies to inform redesign of the query tool. However, CTA does not outperform RTA. The RTA additionally elucidated unique usability problems and new user requirements. Based on the results of this study, we recommend the use of CTA in formative usability evaluation studies of health information technology. However, we recommend further research on the application of RTA in usability studies with regard to user expertise and experience when focusing on user profile customized (re)design. Objective To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes. Methods Sixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method. Results The usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool's screens. RTA detected unique issues concerning system match with subjects' language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians' query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p <0.05) respectively, with an average difference of 42% less time spent per subject compared to RTA. In addition, CTA was more thorough in detecting usability problems of a moderate (0.85 vs. 0.7) and severe nature (0.71 vs. 0.57). Conclusion In this study, the CTA is more effective in usability-problem detection and provided clarification of intensive care physician query design strategies to inform redesign of the query tool. However, CTA does not outperform RTA. The RTA additionally elucidated unique usability problems and new user requirements. Based on the results of this study, we recommend the use of CTA in formative usability evaluation studies of health information technology. However, we recommend further research on the application of RTA in usability studies with regard to user expertise and experience when focusing on user profile customized (re)design. OBJECTIVETo compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes.METHODSSixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method.RESULTSThe usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool's screens. RTA detected unique issues concerning system match with subjects' language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians' query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p<0.05) respectively, with an average difference of 42% less time spent per subject compared to RTA. In addition, CTA was more thorough in detecting usability problems of a moderate (0.85 vs. 0.7) and severe nature (0.71 vs. 0.57).CONCLUSIONIn this study, the CTA is more effective in usability-problem detection and provided clarification of intensive care physician query design strategies to inform redesign of the query tool. However, CTA does not outperform RTA. The RTA additionally elucidated unique usability problems and new user requirements. Based on the results of this study, we recommend the use of CTA in formative usability evaluation studies of health information technology. However, we recommend further research on the application of RTA in usability studies with regard to user expertise and experience when focusing on user profile customized (re)design. |
Author | Jaspers, Monique W.M. de Keizer, Nicolette F. Peute, Linda W.P. |
Author_xml | – sequence: 1 givenname: Linda W.P. surname: Peute fullname: Peute, Linda W.P. email: l.w.peute@amc.uva.nl organization: Academic Medical Center, University of Amsterdam, Department of Medical Informatics, The Netherlands – sequence: 2 givenname: Nicolette F. surname: de Keizer fullname: de Keizer, Nicolette F. email: n.f.keizer@amc.uva.nl organization: Academic Medical Center, University of Amsterdam, Department of Medical Informatics, The Netherlands – sequence: 3 givenname: Monique W.M. surname: Jaspers fullname: Jaspers, Monique W.M. email: m.w.jaspers@amc.uva.nl organization: Academic Medical Center, University of Amsterdam, Department of Medical Informatics, The Netherlands |
BackLink | https://www.ncbi.nlm.nih.gov/pubmed/25733166$$D View this record in MEDLINE/PubMed |
BookMark | eNqNkU1vEzEQhi3Uin7AD-CCfOSSxWPvenfFqYpaWqkSUhXOlteeJQ4bO9jeSPn3dUjpETh5Ds-845nnipz54JGQD8AqYCA_b6rN4CrOoKkYrxiTb8glNIIvWN2xs9da1hfkKqUNYwBNI9-SC960QoCUlySt1kj3epqRhpE-YY4h7dBkt0eqvaXL4M0cI_pMV2vnf9KbKcyWOk_HELf6NzcnPbjJ5QPNmLLzP45Rmu7Wh-SM055anTX9NWMsRAjTO3I-6inh-5f3mny_u10t7xeP374-LG8eF6aWPC-g64QZmg5HYQcDncF-YCBQclHWQ65rY0XN7cAkCit7IzSMIwoh23qAsRfX5NMpdxdDmZ6y2rpkcJq0xzAnBS3rWyElq_8HhbpnjWz_jcquLR_s-66gcEJNuWqKOKpddFsdDwqYOhpUG1UMqqNBxbgqBkvPx5f4ediife34o6wAX04AltPtHUaVjENv0LpYxCkb3F_inwF15q3q |
CitedBy_id | crossref_primary_10_1136_bmjhci_2020_100191 crossref_primary_10_2196_11442 crossref_primary_10_2196_humanfactors_5820 crossref_primary_10_1080_10447318_2020_1809152 crossref_primary_10_1186_s12911_023_02147_x crossref_primary_10_1186_s12911_023_02357_3 crossref_primary_10_3389_fdgth_2022_1020753 crossref_primary_10_2200_S00987ED1V01Y202001HCI045 crossref_primary_10_1186_s12911_020_01280_1 crossref_primary_10_1007_s11121_020_01148_w crossref_primary_10_1016_j_foodqual_2022_104629 crossref_primary_10_1051_e3sconf_202449901010 crossref_primary_10_1891_CRNR_D_20_00092 crossref_primary_10_1007_s11042_021_11042_4 crossref_primary_10_1145_3665327 crossref_primary_10_3390_s21237991 crossref_primary_10_2196_55790 crossref_primary_10_3390_ijerph19063637 crossref_primary_10_3390_s20102750 crossref_primary_10_1016_j_jbi_2016_10_010 crossref_primary_10_1080_13614568_2020_1771436 crossref_primary_10_1007_s12574_021_00531_y crossref_primary_10_3889_oamjms_2021_5914 crossref_primary_10_1016_j_iheduc_2021_100808 crossref_primary_10_1007_s10209_018_0610_z crossref_primary_10_1108_ITP_04_2021_0273 crossref_primary_10_1186_s12911_018_0667_x crossref_primary_10_1016_j_heliyon_2024_e30054 crossref_primary_10_3390_s20185060 crossref_primary_10_2196_21259 |
Cites_doi | 10.1016/j.ijmedinf.2008.07.008 10.1016/j.jbi.2010.11.008 10.1016/j.ijmedinf.2008.06.014 10.1109/47.867942 10.1016/j.jbi.2004.01.003 10.1016/j.ijmedinf.2011.06.010 10.1145/581710.581737 10.1177/154193129003401720 10.1037/0022-0663.90.2.249 10.1080/0044929031000 10.1016/j.jbi.2005.05.002 10.1080/10447318.2012.758529 10.1016/j.ijmedinf.2011.02.005 10.1007/s00134-002-1272-z 10.2307/1423365 10.3758/BF03202637 10.1145/634067.634238 10.1109/47.867941 10.1186/1472-6947-5-37 |
ContentType | Journal Article |
Copyright | 2015 Elsevier Inc. Copyright © 2015 Elsevier Inc. All rights reserved. |
Copyright_xml | – notice: 2015 Elsevier Inc. – notice: Copyright © 2015 Elsevier Inc. All rights reserved. |
DBID | 6I. AAFTH CGR CUY CVF ECM EIF NPM AAYXX CITATION 7X8 7QO 8FD FR3 P64 7SC JQ2 KR7 L7M L~C L~D |
DOI | 10.1016/j.jbi.2015.02.006 |
DatabaseName | ScienceDirect Open Access Titles Elsevier:ScienceDirect:Open Access Medline MEDLINE MEDLINE (Ovid) MEDLINE MEDLINE PubMed CrossRef MEDLINE - Academic Biotechnology Research Abstracts Technology Research Database Engineering Research Database Biotechnology and BioEngineering Abstracts Computer and Information Systems Abstracts ProQuest Computer Science Collection Civil Engineering Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
DatabaseTitle | MEDLINE Medline Complete MEDLINE with Full Text PubMed MEDLINE (Ovid) CrossRef MEDLINE - Academic Engineering Research Database Biotechnology Research Abstracts Technology Research Database Biotechnology and BioEngineering Abstracts Civil Engineering Abstracts Computer and Information Systems Abstracts – Academic ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
DatabaseTitleList | MEDLINE Engineering Research Database MEDLINE - Academic Civil Engineering Abstracts |
Database_xml | – sequence: 1 dbid: NPM name: PubMed url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: EIF name: MEDLINE url: https://proxy.k.utb.cz/login?url=https://www.webofscience.com/wos/medline/basic-search sourceTypes: Index Database |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Medicine Engineering Public Health |
EISSN | 1532-0480 |
EndPage | 10 |
ExternalDocumentID | 10_1016_j_jbi_2015_02_006 25733166 S1532046415000386 |
Genre | Research Support, Non-U.S. Gov't Journal Article Comparative Study |
GroupedDBID | --- --K --M -~X .DC .GJ .~1 0R~ 1B1 1RT 1~. 1~5 29J 4.4 457 4G. 53G 5GY 5VS 6I. 7-5 71M 8P~ AACTN AAEDT AAEDW AAFTH AAIAV AAIKJ AAKOC AALRI AAOAW AAQFI AAQXK AAWTL AAXUO AAYFN ABBOA ABBQC ABFRF ABJNI ABLVK ABMAC ABMZM ABVKL ABXDB ABYKQ ACDAQ ACGFO ACGFS ACNNM ACRLP ACZNC ADBBV ADEZE ADFGL ADMUD AEBSH AEFWE AEKER AENEX AEXQZ AFKWA AFTJW AFXIZ AGHFR AGUBO AGYEJ AHZHX AIALX AIEXJ AIKHN AITUG AJBFU AJOXV AJRQY ALMA_UNASSIGNED_HOLDINGS AMFUW AMRAJ ANZVX AOUOD ASPBG AVWKF AXJTR AZFZN BAWUL BKOJK BLXMC BNPGV CAG COF CS3 DIK DM4 DU5 EBS EFBJH EFLBG EJD EO8 EO9 EP2 EP3 F5P FDB FEDTE FGOYB FIRID FNPLU FYGXN G-Q G8K GBLVA GBOLZ HVGLF HZ~ IHE IXB J1W KOM LCYCR LG5 M41 MO0 N9A NCXOZ O-L O9- OAUVE OK1 OZT P-8 P-9 PC. Q38 R2- RIG ROL RPZ SDF SDG SDP SES SEW SPC SPCBC SSH SSV SSZ T5K UAP UHS UNMZH XPP ZGI ZMT ZU3 ~G- 0SF AAXKI ADVLN AFJKZ AKRWK CGR CUY CVF ECM EIF NPM AAYXX CITATION 7X8 7QO 8FD FR3 P64 7SC JQ2 KR7 L7M L~C L~D |
ID | FETCH-LOGICAL-c462t-1883cb58ef3dbc18ce9b013e623480e2a4cd342db06e3d69c3a1ffe33674b1f93 |
IEDL.DBID | AIKHN |
ISSN | 1532-0464 |
IngestDate | Fri Aug 16 01:55:33 EDT 2024 Fri Aug 16 22:21:04 EDT 2024 Sat Aug 17 02:11:43 EDT 2024 Thu Sep 26 15:56:01 EDT 2024 Sat Sep 28 08:05:18 EDT 2024 Fri Feb 23 02:33:47 EST 2024 |
IsDoiOpenAccess | true |
IsOpenAccess | true |
IsPeerReviewed | true |
IsScholarly | true |
Keywords | Methodologies ICU Databases Think Aloud Usability User–computer interface |
Language | English |
License | http://www.elsevier.com/open-access/userlicense/1.0 Copyright © 2015 Elsevier Inc. All rights reserved. |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-c462t-1883cb58ef3dbc18ce9b013e623480e2a4cd342db06e3d69c3a1ffe33674b1f93 |
Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
OpenAccessLink | https://www.sciencedirect.com/science/article/pii/S1532046415000386 |
PMID | 25733166 |
PQID | 1687348998 |
PQPubID | 23479 |
PageCount | 10 |
ParticipantIDs | proquest_miscellaneous_1709736604 proquest_miscellaneous_1701490567 proquest_miscellaneous_1687348998 crossref_primary_10_1016_j_jbi_2015_02_006 pubmed_primary_25733166 elsevier_sciencedirect_doi_10_1016_j_jbi_2015_02_006 |
PublicationCentury | 2000 |
PublicationDate | June 2015 2015-Jun 2015-06-00 20150601 |
PublicationDateYYYYMMDD | 2015-06-01 |
PublicationDate_xml | – month: 06 year: 2015 text: June 2015 |
PublicationDecade | 2010 |
PublicationPlace | United States |
PublicationPlace_xml | – name: United States |
PublicationTitle | Journal of biomedical informatics |
PublicationTitleAlternate | J Biomed Inform |
PublicationYear | 2015 |
Publisher | Elsevier Inc |
Publisher_xml | – name: Elsevier Inc |
References | Norio Ishii, Kazuhisa Miwa. Interactive processes between mental and external operations in creative activity: a comparison of experts’ and novices’ performance. In: Proceedings of the 4th conference on creativity & cognition. 2002 Oct. 13–16. p. 178–85. Kohn LT et al. editors. To Err is human: building a safer health system. Institute of Medicine. Dix, Finlay, Abowd, Beale (b0030) 2003 2010 Sears (b0130) 1997; 9 Spires, Donley (b0175) 1998; 90 Hartson, Andre, Williges (b0100) 2001; 13 Guan, Lee, Cuddihy, Ramey (b0200) 2006 Horsky, Zhang, Patel (b0020) 2005; 38 De Keizer, Peute, Van der Zwan, Jaspers, De Jonge (b0110) 2011 June; 15 Svanæs, Alsos, Dahl (b0035) 2010; 79 Norman (b0145) 1988 Nielsen J. Getting access to what goes on in people’s heads?: reflections on the think-aloud technique. NordiCHI ‘02 Proceedings of the second Nordic conference on Human-computer interaction, p: 101–110. Johnson-Laird (b0160) 1983 Khajouei, Hasman, Jaspers (b0045) 2011; 80 De Jong, Schellens (b0085) 2000; 43 Shertz, Weiser (b0180) 1981 Arts, de Keizer, Scheffer, de Jonge (b0105) 2002; 28 Boren, Ramey (b0070) 2000; 43 Kuusela, Paul (b0155) 2000; 113 Clayton LH. Using the think aloud method in cognitive interface design. IBM RC-9265.1982. Yorktown heights. NY: IBM Thomas J Watson Research Center. Ericsson, Simon (b0065) 1993 Payne (b0140) 2003 Nielsen (b0195) 1989 Nielsen J. Novice vs. expert users. Alertbox, February 6, 2000. Haak, van den, Jong de, Schellens (b0165) 2003; 22 Muin, Fontelo, Liu, Ackerman (b0185) 2005 Dec; 1 ISO 9241-11. International standard: ISO 9241 ergonomic requirements for office work with visual display terminals, Part 11: Guidance on usability. Geneve: International Organization for Standardization; 1996. Turner, Lewis, Nielsen (b0135) 2006; vol. 3, 2nd ed. McDonald, Zhao, Edwards (b0170) 2013; 29 Russo, Johnson, Stephens (b0090) 1989; 17 Horvath, Winfield, Evans, Slopek, Shang, Ferranti (b0190) 2011; 44 . Bowers VA, Snyder HL. Concurrent versus retrospective verbal protocols for comparing window usability. Human factors society 34th meeting. 1990 Oct. 8–12 (Santa Monica: HFES). p. 1270–1274. Kushniruk, Patel (b0115) 2004; 37 Nielsen J. Severity ratings for usability problems. Teague R, De Jesus K, Nunes-Ueno M. Concurrent vs. post-task usability test ratings. In: Conference on human factors and computing systems, 2001 31 March–5 April (Seattle, WA: ACM SIGCHI):289–290. Campbell EM, Sittig DF. Guappone KP, Dykstra RH, Ash JS. Overdependence on technology: an unintended adverse consequence of computerized provider order entry. In: AMIA annu symp proc. 2007. p. 94–8. National questionnaire study on clinical ICT systems proofs: Physicians suffer from poor usability. Johanna Viitanen, Hannele Hyppönen, Tinja Lääveri, Jukka Vänskä, Jarmo Reponen, Ilkka Winblad. Int J Med Inf 2011; 80(10):708–25. [cited 10 A.D. Oct 24 2011]. Lai (b0040) 2007; 76 Kjeldskov, Skov, Stage (b0060) 2010; 79 Gwet KL. Handbook of inter-rater reliability, 2nd ed.; 2010. (Gaithersburg: Advanced Analytics, LLC) ISBN 978-0970806222. Horsky (10.1016/j.jbi.2015.02.006_b0020) 2005; 38 Ericsson (10.1016/j.jbi.2015.02.006_b0065) 1993 Horvath (10.1016/j.jbi.2015.02.006_b0190) 2011; 44 10.1016/j.jbi.2015.02.006_b0005 10.1016/j.jbi.2015.02.006_b0125 De Jong (10.1016/j.jbi.2015.02.006_b0085) 2000; 43 Russo (10.1016/j.jbi.2015.02.006_b0090) 1989; 17 Spires (10.1016/j.jbi.2015.02.006_b0175) 1998; 90 Arts (10.1016/j.jbi.2015.02.006_b0105) 2002; 28 10.1016/j.jbi.2015.02.006_b0050 10.1016/j.jbi.2015.02.006_b0010 10.1016/j.jbi.2015.02.006_b0075 Lai (10.1016/j.jbi.2015.02.006_b0040) 2007; 76 Guan (10.1016/j.jbi.2015.02.006_b0200) 2006 10.1016/j.jbi.2015.02.006_b0095 10.1016/j.jbi.2015.02.006_b0150 Johnson-Laird (10.1016/j.jbi.2015.02.006_b0160) 1983 Shertz (10.1016/j.jbi.2015.02.006_b0180) 1981 10.1016/j.jbi.2015.02.006_b0055 Kjeldskov (10.1016/j.jbi.2015.02.006_b0060) 2010; 79 Turner (10.1016/j.jbi.2015.02.006_b0135) 2006; vol. 3, 2nd ed. Khajouei (10.1016/j.jbi.2015.02.006_b0045) 2011; 80 10.1016/j.jbi.2015.02.006_b0015 Haak, van den (10.1016/j.jbi.2015.02.006_b0165) 2003; 22 De Keizer (10.1016/j.jbi.2015.02.006_b0110) 2011; 15 Hartson (10.1016/j.jbi.2015.02.006_b0100) 2001; 13 Kuusela (10.1016/j.jbi.2015.02.006_b0155) 2000; 113 Nielsen (10.1016/j.jbi.2015.02.006_b0195) 1989 McDonald (10.1016/j.jbi.2015.02.006_b0170) 2013; 29 Norman (10.1016/j.jbi.2015.02.006_b0145) 1988 Sears (10.1016/j.jbi.2015.02.006_b0130) 1997; 9 Boren (10.1016/j.jbi.2015.02.006_b0070) 2000; 43 Payne (10.1016/j.jbi.2015.02.006_b0140) 2003 Muin (10.1016/j.jbi.2015.02.006_b0185) 2005; 1 10.1016/j.jbi.2015.02.006_b0080 Kushniruk (10.1016/j.jbi.2015.02.006_b0115) 2004; 37 Dix (10.1016/j.jbi.2015.02.006_b0030) 2003 Svanæs (10.1016/j.jbi.2015.02.006_b0035) 2010; 79 10.1016/j.jbi.2015.02.006_b0120 10.1016/j.jbi.2015.02.006_b0025 |
References_xml | – volume: 37 start-page: 56 year: 2004 end-page: 76 ident: b0115 article-title: Cognitive and usability engineering methods for the evaluation of clinical information systems publication-title: J Biomed Inform contributor: fullname: Patel – volume: 43 start-page: 261 year: 2000 end-page: 278 ident: b0070 article-title: Thinking aloud: reconciling theory and practice publication-title: IEEE Trans Prof Comm contributor: fullname: Ramey – volume: 90 start-page: 249 year: 1998 end-page: 260 ident: b0175 article-title: Prior knowledge activation: inducing engagement with informational texts publication-title: J Educ Psychol contributor: fullname: Donley – volume: 44 start-page: 266 year: 2011 end-page: 276 ident: b0190 article-title: The DEDUCE Guided Query tool: Providing simplified access to clinical data for research and quality improvement publication-title: J Biomed Inform contributor: fullname: Ferranti – volume: 80 start-page: 341 year: 2011 end-page: 350 ident: b0045 article-title: Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system publication-title: Int J Med Inform contributor: fullname: Jaspers – start-page: 1253 year: 2006 end-page: 1262 ident: b0200 article-title: The validity of the stimulated retrospective think-aloud method as measured by eye tracking publication-title: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ‘06) contributor: fullname: Ramey – year: 1988 ident: b0145 article-title: The design of everyday things contributor: fullname: Norman – volume: 79 start-page: e24 year: 2010 end-page: e34 ident: b0035 article-title: Usability testing of mobile ICT for clinical settings: methodological and practical challenges publication-title: Int J Med Inform contributor: fullname: Dahl – volume: 76 start-page: 2 year: 2007 ident: b0040 article-title: Iterative refinement of a tailored system for self-care management of depressive symptoms in people living with HIV/AIDS through heuristic evaluation and end user testing publication-title: Int J Med Inform contributor: fullname: Lai – volume: 9 start-page: 213 year: 1997 end-page: 234 ident: b0130 article-title: Heuristic walkthroughs: finding the problems without the noise publication-title: IJHCI contributor: fullname: Sears – year: 2003 ident: b0030 article-title: Human–computer interaction contributor: fullname: Beale – year: 1993 ident: b0065 article-title: Protocol analysis: verbal protocols as data contributor: fullname: Simon – start-page: 302 year: 1981 end-page: 322 ident: b0180 article-title: A study of programming problem representation in novice and expert programmers publication-title: Proceedings of the eighteenth annual computer personnel research conference (SIGCPR ‘81) contributor: fullname: Weiser – volume: 1 start-page: 37 year: 2005 Dec ident: b0185 article-title: SLIM: an alternative Web interface for MEDLINE/PubMed searches – a preliminary study publication-title: BMC Med Inform Decis Making contributor: fullname: Ackerman – volume: 13 start-page: 241 year: 2001 end-page: 444 ident: b0100 article-title: Criteria for evaluating usability evaluation methods publication-title: Int J Hum Compos Interact contributor: fullname: Williges – volume: 15 start-page: 131 year: 2011 June end-page: 136 ident: b0110 article-title: NICE online; a web-based tool for monitoring performance measures in intensive care publication-title: Neth J Crit Care contributor: fullname: De Jonge – year: 1983 ident: b0160 article-title: Mental models: towards a cognitive science of language, inference, and consciousness contributor: fullname: Johnson-Laird – start-page: 394 year: 1989 end-page: 401 ident: b0195 article-title: Usability engineering at a discount publication-title: Designing and using human–computer interfaces and knowledge based systems contributor: fullname: Nielsen – volume: 113 start-page: 387 year: 2000 end-page: 404 ident: b0155 article-title: A comparison of concurrent and retrospective verbal protocol analysis publication-title: Am J Psychol contributor: fullname: Paul – volume: 17 start-page: 759 year: 1989 end-page: 769 ident: b0090 article-title: The validity of verbal protocols publication-title: Mem Cognition contributor: fullname: Stephens – volume: 38 start-page: 264 year: 2005 end-page: 266 ident: b0020 article-title: To err is not entirely human: complex technology and user cognition publication-title: J Biomed Inform contributor: fullname: Patel – start-page: 135 year: 2003 end-page: 154 ident: b0140 article-title: Users’ mental models: the very ideas publication-title: HCI models, theories, and frameworks contributor: fullname: Payne – volume: 29 year: 2013 ident: b0170 article-title: Dual verbal elicitation: the complementary use of concurrent and retrospective reporting within a usability test publication-title: Int J Human Comput Interact contributor: fullname: Edwards – volume: 28 start-page: 656 year: 2002 end-page: 659 ident: b0105 article-title: Quality of data collected for severity of illness scores in the Dutch National Intensive Care Evaluation (NICE) registry publication-title: Intens Care Med contributor: fullname: de Jonge – volume: 22 start-page: 339 year: 2003 end-page: 351 ident: b0165 article-title: Retrospective versus concurrent think-aloud protocols: testing the usability of an online library catalogue publication-title: Behav Inform Technol contributor: fullname: Schellens – volume: 79 start-page: e135 year: 2010 end-page: e143 ident: b0060 article-title: A longitudinal study of usability in health care: does time heal? publication-title: Int J Med Inform contributor: fullname: Stage – volume: 43 start-page: 242 year: 2000 end-page: 260 ident: b0085 article-title: Towards a document evaluation methodology: what does research tell us about the validity and reliability of methods? publication-title: IEEE Trans Prof Commun contributor: fullname: Schellens – volume: vol. 3, 2nd ed. year: 2006 ident: b0135 article-title: Determining usability test sample size publication-title: International encyclopedia of ergonomics and human factors contributor: fullname: Nielsen – start-page: 302 year: 1981 ident: 10.1016/j.jbi.2015.02.006_b0180 article-title: A study of programming problem representation in novice and expert programmers contributor: fullname: Shertz – ident: 10.1016/j.jbi.2015.02.006_b0075 – ident: 10.1016/j.jbi.2015.02.006_b0025 – volume: 79 start-page: e135 issue: 6 year: 2010 ident: 10.1016/j.jbi.2015.02.006_b0060 article-title: A longitudinal study of usability in health care: does time heal? publication-title: Int J Med Inform doi: 10.1016/j.ijmedinf.2008.07.008 contributor: fullname: Kjeldskov – year: 2003 ident: 10.1016/j.jbi.2015.02.006_b0030 contributor: fullname: Dix – volume: 44 start-page: 266 issue: 2 year: 2011 ident: 10.1016/j.jbi.2015.02.006_b0190 article-title: The DEDUCE Guided Query tool: Providing simplified access to clinical data for research and quality improvement publication-title: J Biomed Inform doi: 10.1016/j.jbi.2010.11.008 contributor: fullname: Horvath – volume: 79 start-page: e24 year: 2010 ident: 10.1016/j.jbi.2015.02.006_b0035 article-title: Usability testing of mobile ICT for clinical settings: methodological and practical challenges publication-title: Int J Med Inform doi: 10.1016/j.ijmedinf.2008.06.014 contributor: fullname: Svanæs – volume: 43 start-page: 261 year: 2000 ident: 10.1016/j.jbi.2015.02.006_b0070 article-title: Thinking aloud: reconciling theory and practice publication-title: IEEE Trans Prof Comm doi: 10.1109/47.867942 contributor: fullname: Boren – volume: 13 start-page: 241 issue: 4 year: 2001 ident: 10.1016/j.jbi.2015.02.006_b0100 article-title: Criteria for evaluating usability evaluation methods publication-title: Int J Hum Compos Interact contributor: fullname: Hartson – volume: vol. 3, 2nd ed. year: 2006 ident: 10.1016/j.jbi.2015.02.006_b0135 article-title: Determining usability test sample size contributor: fullname: Turner – volume: 37 start-page: 56 year: 2004 ident: 10.1016/j.jbi.2015.02.006_b0115 article-title: Cognitive and usability engineering methods for the evaluation of clinical information systems publication-title: J Biomed Inform doi: 10.1016/j.jbi.2004.01.003 contributor: fullname: Kushniruk – ident: 10.1016/j.jbi.2015.02.006_b0015 doi: 10.1016/j.ijmedinf.2011.06.010 – ident: 10.1016/j.jbi.2015.02.006_b0050 doi: 10.1145/581710.581737 – ident: 10.1016/j.jbi.2015.02.006_b0080 doi: 10.1177/154193129003401720 – start-page: 394 year: 1989 ident: 10.1016/j.jbi.2015.02.006_b0195 article-title: Usability engineering at a discount contributor: fullname: Nielsen – volume: 90 start-page: 249 issue: 2 year: 1998 ident: 10.1016/j.jbi.2015.02.006_b0175 article-title: Prior knowledge activation: inducing engagement with informational texts publication-title: J Educ Psychol doi: 10.1037/0022-0663.90.2.249 contributor: fullname: Spires – volume: 22 start-page: 339 issue: 5 year: 2003 ident: 10.1016/j.jbi.2015.02.006_b0165 article-title: Retrospective versus concurrent think-aloud protocols: testing the usability of an online library catalogue publication-title: Behav Inform Technol doi: 10.1080/0044929031000 contributor: fullname: Haak, van den – start-page: 135 year: 2003 ident: 10.1016/j.jbi.2015.02.006_b0140 article-title: Users’ mental models: the very ideas contributor: fullname: Payne – ident: 10.1016/j.jbi.2015.02.006_b0120 – volume: 38 start-page: 264 issue: 4 year: 2005 ident: 10.1016/j.jbi.2015.02.006_b0020 article-title: To err is not entirely human: complex technology and user cognition publication-title: J Biomed Inform doi: 10.1016/j.jbi.2005.05.002 contributor: fullname: Horsky – year: 1988 ident: 10.1016/j.jbi.2015.02.006_b0145 contributor: fullname: Norman – volume: 29 issue: 10 year: 2013 ident: 10.1016/j.jbi.2015.02.006_b0170 article-title: Dual verbal elicitation: the complementary use of concurrent and retrospective reporting within a usability test publication-title: Int J Human Comput Interact doi: 10.1080/10447318.2012.758529 contributor: fullname: McDonald – volume: 80 start-page: 341 issue: 5 year: 2011 ident: 10.1016/j.jbi.2015.02.006_b0045 article-title: Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system publication-title: Int J Med Inform doi: 10.1016/j.ijmedinf.2011.02.005 contributor: fullname: Khajouei – volume: 28 start-page: 656 issue: 5 year: 2002 ident: 10.1016/j.jbi.2015.02.006_b0105 article-title: Quality of data collected for severity of illness scores in the Dutch National Intensive Care Evaluation (NICE) registry publication-title: Intens Care Med doi: 10.1007/s00134-002-1272-z contributor: fullname: Arts – volume: 113 start-page: 387 issue: 3 year: 2000 ident: 10.1016/j.jbi.2015.02.006_b0155 article-title: A comparison of concurrent and retrospective verbal protocol analysis publication-title: Am J Psychol doi: 10.2307/1423365 contributor: fullname: Kuusela – ident: 10.1016/j.jbi.2015.02.006_b0055 – volume: 17 start-page: 759 year: 1989 ident: 10.1016/j.jbi.2015.02.006_b0090 article-title: The validity of verbal protocols publication-title: Mem Cognition doi: 10.3758/BF03202637 contributor: fullname: Russo – ident: 10.1016/j.jbi.2015.02.006_b0005 – ident: 10.1016/j.jbi.2015.02.006_b0150 – ident: 10.1016/j.jbi.2015.02.006_b0095 doi: 10.1145/634067.634238 – volume: 43 start-page: 242 year: 2000 ident: 10.1016/j.jbi.2015.02.006_b0085 article-title: Towards a document evaluation methodology: what does research tell us about the validity and reliability of methods? publication-title: IEEE Trans Prof Commun doi: 10.1109/47.867941 contributor: fullname: De Jong – volume: 76 start-page: 2 issue: SUPPL year: 2007 ident: 10.1016/j.jbi.2015.02.006_b0040 article-title: Iterative refinement of a tailored system for self-care management of depressive symptoms in people living with HIV/AIDS through heuristic evaluation and end user testing publication-title: Int J Med Inform contributor: fullname: Lai – start-page: 1253 year: 2006 ident: 10.1016/j.jbi.2015.02.006_b0200 article-title: The validity of the stimulated retrospective think-aloud method as measured by eye tracking contributor: fullname: Guan – volume: 15 start-page: 131 issue: 3 year: 2011 ident: 10.1016/j.jbi.2015.02.006_b0110 article-title: NICE online; a web-based tool for monitoring performance measures in intensive care publication-title: Neth J Crit Care contributor: fullname: De Keizer – volume: 9 start-page: 213 year: 1997 ident: 10.1016/j.jbi.2015.02.006_b0130 article-title: Heuristic walkthroughs: finding the problems without the noise publication-title: IJHCI contributor: fullname: Sears – year: 1983 ident: 10.1016/j.jbi.2015.02.006_b0160 contributor: fullname: Johnson-Laird – year: 1993 ident: 10.1016/j.jbi.2015.02.006_b0065 contributor: fullname: Ericsson – ident: 10.1016/j.jbi.2015.02.006_b0125 – ident: 10.1016/j.jbi.2015.02.006_b0010 – volume: 1 start-page: 37 issue: 5 year: 2005 ident: 10.1016/j.jbi.2015.02.006_b0185 article-title: SLIM: an alternative Web interface for MEDLINE/PubMed searches – a preliminary study publication-title: BMC Med Inform Decis Making doi: 10.1186/1472-6947-5-37 contributor: fullname: Muin |
SSID | ssj0011556 |
Score | 2.3542347 |
Snippet | [Display omitted]
•Comparison of Concurrent and Retrospective Think Aloud in usability testing of HIT.•Concurrent and Retrospective Think Aloud differ in... To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of... OBJECTIVETo compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability... Objective To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability... |
SourceID | proquest crossref pubmed elsevier |
SourceType | Aggregation Database Index Database Publisher |
StartPage | 1 |
SubjectTerms | Attitude of Health Personnel Consumer Behavior - statistics & numerical data Data Mining - classification Data Mining - methods Data Mining - statistics & numerical data Design engineering Electronic Health Records - statistics & numerical data Error detection ICU Meaningful Use - statistics & numerical data Methodologies Physicians Practice Patterns, Physicians' - classification Practice Patterns, Physicians' - statistics & numerical data Query processing Retrospective Studies Software Software Validation Strategy Symbols Think Aloud Usability User requirements User–computer interface Utilization Review - methods |
Title | The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool |
URI | https://dx.doi.org/10.1016/j.jbi.2015.02.006 https://www.ncbi.nlm.nih.gov/pubmed/25733166 https://search.proquest.com/docview/1687348998 https://search.proquest.com/docview/1701490567 https://search.proquest.com/docview/1709736604 |
Volume | 55 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1Lb9NAEB61qYSoEILwaHhEi8QJycT2rsf2MQ1UCaE9AEW5rbwPo5TKrlLnwIXfzqzXjuDQHLhY8mpteXfG855vAd6KHI21WR6k3GAghAs0FYUJcmVTdJ2dWrvm5PMLnF-KT6tkdQCzvhfGlVV2st_L9FZadyOTbjcnN-v15GvkzjQQSBqozW_hIRyROhJiAEfTxXJ-sUsmkMpED5sau0pG0Sc32zKvK7V2BV6JR-7Eu9TTXeZnq4bOHsHDzn5kU_-Jj-HAVkM4_gtVcAj3zrt8-RAe-Kgc881GT-CWuII5fG_L6pJ9sc2m7lstWVEZNqsr7QGbmDvQ8yebXtdbw9YV62xbmrf1oLzNL9Y4hI7qh3tVwXYhEuaKThmtYkMz6vr6KVyeffw2mwfdsQuBFhg3QZRlXKsksyU3SkeZtnkbLCVDSWShjQuhDRexUSFaInCueRGVpeUcU6GiMufPYFDVlT0BZklEJNyUmnxeYULMDUcTqiLPEoXkbI7gXb_b8saja8i-7OxKEmmkI40MY0mkGYHo6SH_YRFJ0n_fY2962kn6dVw-pKhsvb2VEWYO24cczj1zUudDkpWY7p2TpxwxFCN47pljt5rY4U1GiC_-7-Nfwn1352vTXsGg2Wzta7KCGjWGw_e_ozHx-un35edxx_M0-mGxpOtidfoHguoI-A |
link.rule.ids | 315,786,790,3525,4521,24144,27600,27955,27956,45618,45696,45712,45907 |
linkProvider | Elsevier |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1Lb9QwEB6VIvEQQrBA2ZaHkTghRZvEziQ5lhXVFro9QCvtzUpsp9pSJdU2e-i_70ycrODQPfSaTBLbMxnP8zPAV5WjdS7Lg1RaDJTiQFNR2CAvXYrc2WkMNyfPT3F2rn4uksUOTIdeGC6r7HW_1-mdtu6vTPrVnFwvl5M_EZ9poJB2oC6_hY_gMX0qZA_sePF9k0qgDRM9aGrMdYxqSG12RV6X5ZLLuxKP24n3bU73GZ_dJnT0Cl721qM49AN8DTuuHsHzfzAFR_Bk3mfLR_DCx-SEbzV6AzckE4LRvZ1oKvHbtatmaLQURW3FtKmNh2sSfJznX3F41aytWNait2yJbu0hedtb0TI-R33BryrEJkAiuORU0CxWRNE0V2_h_OjH2XQW9IcuBEZh3AZRlklTJpmrpC1NlBmXd6FSMpNUFrq4UMZKFdsyREfszY0soqpyUmKqyqjK5TvYrZvavQfhSEEk0laGPF5lQ8ytRBuWRZ4lJZKrOYZvw2rra4-toYeis0tNrNHMGh3GmlgzBjXwQ_8nIJp0_7bHvgy80_TjcDakqF2zvtERZozsQ-7mFpqUPUiyEdOtNHkqEUM1hj0vHJvZxIw2GSHuP2zwn-Hp7Gx-ok-OT38dwDO-46vUPsBuu1q7j2QPteWnTt7vAFwvBko |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+value+of+Retrospective+and+Concurrent+Think+Aloud+in+formative+usability+testing+of+a+physician+data+query+tool&rft.jtitle=Journal+of+biomedical+informatics&rft.au=Peute%2C+Linda+WP&rft.au=de+Keizer%2C+Nicolette+F&rft.au=Jaspers%2C+Monique+WM&rft.date=2015-06-01&rft.issn=1532-0464&rft.volume=55&rft.spage=1&rft.epage=10&rft_id=info:doi/10.1016%2Fj.jbi.2015.02.006&rft.externalDBID=NO_FULL_TEXT |
thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1532-0464&client=summon |
thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1532-0464&client=summon |
thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1532-0464&client=summon |