High variability in Summative Usability Test methods & reporting among clinical informatics vendors complying with Federal certification requirements

The Office of the National Coordinator for Health Information Technology (ONC) has been tasked with managing the integration of usability best practices into electronic health record development but, has provided little guidance on how to conduct Summative Usability Testing (SUT). We reviewed three...

Full description

Saved in:
Bibliographic Details
Published inProceedings of the Human Factors and Ergonomics Society Annual Meeting Vol. 58; no. 1; pp. 1491 - 1495
Main Authors Buchanan, Christine, Threatt, Anthony, Weinger, Matthew B., Miller, Anne
Format Journal Article
LanguageEnglish
Published Los Angeles, CA SAGE Publications 01.09.2014
Online AccessGet full text

Cover

Abstract The Office of the National Coordinator for Health Information Technology (ONC) has been tasked with managing the integration of usability best practices into electronic health record development but, has provided little guidance on how to conduct Summative Usability Testing (SUT). We reviewed three aspects of SUT in four publicly available SUT vendor reports. Significant deviations from human factors engineering (HFE) and user-centered design (UCD) best practices and clear methodological errors were found in all four reports. The main sources of variation involved non-representative participant selection, instructions that led participant performance, and varied interpretation of results reporting requirements. Although ONC has made significant gains in implementing best practice UCD approaches, more explicit guidance may be needed to enhance integrity and reduce variation.
AbstractList The Office of the National Coordinator for Health Information Technology (ONC) has been tasked with managing the integration of usability best practices into electronic health record development but, has provided little guidance on how to conduct Summative Usability Testing (SUT). We reviewed three aspects of SUT in four publicly available SUT vendor reports. Significant deviations from human factors engineering (HFE) and user-centered design (UCD) best practices and clear methodological errors were found in all four reports. The main sources of variation involved non-representative participant selection, instructions that led participant performance, and varied interpretation of results reporting requirements. Although ONC has made significant gains in implementing best practice UCD approaches, more explicit guidance may be needed to enhance integrity and reduce variation.
Author Buchanan, Christine
Threatt, Anthony
Miller, Anne
Weinger, Matthew B.
Author_xml – sequence: 1
  givenname: Christine
  surname: Buchanan
  fullname: Buchanan, Christine
  organization: Center for Research and Innovation in Systems Safety, Vanderbilt University Medical Center, Nashville, TN 37212
– sequence: 2
  givenname: Anthony
  surname: Threatt
  fullname: Threatt, Anthony
  organization: Center for Research and Innovation in Systems Safety, Vanderbilt University Medical Center, Nashville, TN 37212
– sequence: 3
  givenname: Matthew B.
  surname: Weinger
  fullname: Weinger, Matthew B.
  organization: Center for Research and Innovation in Systems Safety, Vanderbilt University Medical Center, Nashville, TN 37212
– sequence: 4
  givenname: Anne
  surname: Miller
  fullname: Miller, Anne
  organization: Center for Research and Innovation in Systems Safety, Vanderbilt University Medical Center, Nashville, TN 37212
BookMark eNp1kDFPwzAQhS1UJFpgZ_TEFsglceKMqKKAVIkBOkeOc25dJXax3aL-EP4vjgoLEsuddO99T6c3IxNjDRJyA-kdQFXdAyugziGDgnHIAc7INIOyTlhaVhMyHeVk1C_IzPttmmZ5lRdT8vWs1xt6EE6LVvc6HKk29G0_DCLoA9KV_z2_ow90wLCxnae31OHOuqDNmorBxil7bbQUfcSVdSMtPT2g6azzVNph1x9H86cOG7rADl20SowJKlJBWxMTP_ba4YAm-CtyrkTv8fpnX5LV4vF9_pwsX59e5g_LREJdh0RmTCiWi5RxiQXHXLU1L9ou50p2ZSmhEzVjquPYcgDFCiV4VXEsRVm1KmP5JUlPudJZ7x2qZuf0INyxgbQZa23-1hqR5IR4scZma_fOxA__938Dm7x-VQ
Cites_doi 10.1001/archinternmed.2008.542
10.1518/001872006776412234
10.1177/154193120705101119
10.1001/jama.1997.03540280045032
10.1001/archinte.165.10.1111
10.1001/jama.1995.03530010049034
10.1001/jama.2010.1497
10.1001/jama.293.10.1197
ContentType Journal Article
Copyright 2014 Human Factors and Ergonomics Society
Copyright_xml – notice: 2014 Human Factors and Ergonomics Society
DBID AAYXX
CITATION
DOI 10.1177/1541931214581311
DatabaseName CrossRef
DatabaseTitle CrossRef
DatabaseTitleList
CrossRef
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISSN 2169-5067
EndPage 1495
ExternalDocumentID 10_1177_1541931214581311
10.1177_1541931214581311
GroupedDBID -MK
-TM
.2G
.2L
.2N
01A
09Z
0R~
1~K
29P
3V.
4.4
54M
85S
88I
8AF
8FI
8FJ
8R4
8R5
AABOD
AACKU
AACTG
AADIR
AADUE
AAGGD
AAJOX
AAJPV
AAKTJ
AAMFR
AAMGE
AANSI
AAPEO
AARIX
AATAA
AATBZ
AAWLO
AAYTG
ABAWP
ABCCA
ABDWY
ABEIX
ABFWQ
ABFXH
ABHKI
ABKRH
ABLUO
ABPNF
ABQKF
ABQPY
ABQXT
ABRHV
ABUWG
ABYTW
ACAEP
ACDXX
ACFUR
ACFZE
ACGBL
ACGFS
ACGOD
ACLZU
ACOFE
ACOXC
ACROE
ACSIQ
ACTQU
ACUAV
ACUFS
ACUIR
ACXKE
ADBBV
ADEIA
ADPEE
ADRRZ
ADTBJ
ADTOS
ADUKL
AEOBU
AEPTA
AEQLS
AESMA
AESZF
AEUHG
AEUIJ
AEVPJ
AEWDL
AEWHI
AEXNY
AFEET
AFKBI
AFKRA
AFKRG
AFMOU
AFQAA
AFUIA
AGDVU
AGKLV
AGNHF
AGNWV
AGWFA
AHWHD
AIOMO
AJEFB
AJUZI
ALFTD
ALJHS
ALMA_UNASSIGNED_HOLDINGS
ARTOV
AUVAJ
AYPQM
AZFZN
AZQEC
BBRGL
BDDNI
BENPR
BMVBW
BPACV
BPHCQ
BVXVI
BYIEH
CBRKF
CCGJY
CCPQU
CEADM
CFDXU
CORYS
CS3
DD0
DD~
DE-
DG~
DO-
DOPDO
DV7
DV8
DWQXO
D~Y
EBS
EJD
FHBDP
FYUFA
GNUQQ
GROUPED_SAGE_PREMIER_JOURNAL_COLLECTION
HCIFZ
HF~
HVGLF
J8X
JCYGO
K.F
KQ4
M2M
M2P
M4V
O9-
P.B
PQQKQ
PROAC
PSYQQ
Q1R
Q2X
Q7O
Q7P
Q7V
Q7X
Q82
Q83
ROL
S01
SCNPE
SFC
SPV
SSDHQ
UKHRP
ZPLXX
ZPPRI
ZRKOI
~32
AAYXX
AEDXQ
ANDLU
CITATION
H13
ID FETCH-LOGICAL-c199t-c25af53a058ce48e3fb984bd38fcd66c1da955fd8eb811f54fa8778e6a67bf253
ISSN 1541-9312
1071-1813
IngestDate Wed Oct 02 14:37:41 EDT 2024
Tue Jul 16 20:47:19 EDT 2024
IsPeerReviewed true
IsScholarly true
Issue 1
Language English
LinkModel OpenURL
MergedId FETCHMERGED-LOGICAL-c199t-c25af53a058ce48e3fb984bd38fcd66c1da955fd8eb811f54fa8778e6a67bf253
PageCount 5
ParticipantIDs crossref_primary_10_1177_1541931214581311
sage_journals_10_1177_1541931214581311
PublicationCentury 2000
PublicationDate 20140900
PublicationDateYYYYMMDD 2014-09-01
PublicationDate_xml – month: 9
  year: 2014
  text: 20140900
PublicationDecade 2010
PublicationPlace Los Angeles, CA
PublicationPlace_xml – name: Los Angeles, CA
PublicationTitle Proceedings of the Human Factors and Ergonomics Society Annual Meeting
PublicationYear 2014
Publisher SAGE Publications
Publisher_xml – name: SAGE Publications
References Bates, Spell, Cullen, Burdick, Laird, Petersen, Small 1997; 277
Patterson, Rogers, Chapman, Render 2006; 48
Bates 2009; 169
Campbell, Sittig, Ash, Guappone, Dykstra 2006; 13
Jha 2010; 304
Koppel, Metlay, Cohen, Abaluck, Localio, Kimmel, Strom 2005; 293
Leape, Bates, Cullen, Cooper, Demonaco, Gallivan, Hallisey, Ives 1995; 274
Koppel, Wetterneck, Telles, Karsh 2008; 15
Karsh, Weinger, Abbott, Wears 2010; 17
Nebecker, Hoffman, Weir, Bennett, Hurcle 2005; 165
Kuperman, Teich, Gandhi, Bates 2001; 27
Karsh BT (bibr7-1541931214581311) 2010; 17
bibr11-1541931214581311
bibr13-1541931214581311
Campbell EM (bibr3-1541931214581311) 2006; 13
International Organisation for Standardisation (bibr4-1541931214581311) 2010
Koppel R (bibr9-1541931214581311) 2008; 15
Stone D (bibr17-1541931214581311) 2005
Rubin J (bibr15-1541931214581311) 2008
bibr2-1541931214581311
Schumacher R (bibr16-1541931214581311) 2012
bibr6-1541931214581311
bibr12-1541931214581311
bibr8-1541931214581311
Kuperman G. J. (bibr10-1541931214581311) 2001; 27
bibr14-1541931214581311
Israelski E (bibr5-1541931214581311) 2011
bibr1-1541931214581311
References_xml – volume: 27
  start-page: 509
  issue: 10
  year: 2001
  end-page: 521
  article-title: Patient safety and computerized medication ordering at Brigham and Women’s Hospital
  publication-title: Joint Commission Journal on Quality and Patient Safety
  contributor:
    fullname: Bates
– volume: 274
  start-page: 35
  year: 1995
  end-page: 43
  article-title: Systems analysis of adverse drug events
  publication-title: JAMA
  contributor:
    fullname: Ives
– volume: 13
  start-page: 547
  year: 2006
  end-page: 556
  article-title: Types of unintended consequences related to computerized provider order entry
  publication-title: JAMIA
  contributor:
    fullname: Dykstra
– volume: 304
  start-page: 10
  issue: 15
  year: 2010
  end-page: 20
  article-title: Meaningful use of electronic health records: The road ahead
  publication-title: JAMA
  contributor:
    fullname: Jha
– volume: 48
  start-page: 15
  issue: 1
  year: 2006
  end-page: 22
  article-title: compliance with intended use of barcode medication Administration in acute and long-term care: An observational study
  publication-title: Human Factors
  contributor:
    fullname: Render
– volume: 165
  start-page: 1111
  year: 2005
  end-page: 6
  article-title: High rates of adverse drug events in a highly computerized hospital
  publication-title: Arch Intern Med
  contributor:
    fullname: Hurcle
– volume: 277
  start-page: 307
  year: 1997
  end-page: 11
  article-title: The costs of adverse drug events in hospitalized patients
  publication-title: JAMA
  contributor:
    fullname: Small
– volume: 169
  start-page: 1
  issue: 2
  year: 2009
  end-page: 26
  article-title: The effects of health information technology on inpatient care
  publication-title: Arch Intern Med
  contributor:
    fullname: Bates
– volume: 17
  start-page: 617
  year: 2010
  end-page: 23
  article-title: Health Information technology: Fallacies and sober realities
  publication-title: JAMIA
  contributor:
    fullname: Wears
– volume: 15
  start-page: 408
  year: 2008
  end-page: 423
  article-title: Workarounds to barcode medication administration systems: Their occurrences, causes, and threats to patient safety
  publication-title: JAMIA
  contributor:
    fullname: Karsh
– volume: 293
  start-page: 1197
  year: 2005
  end-page: 203
  article-title: Role of computerized physician order entry systems in facilitating medication errors
  publication-title: JAMA
  contributor:
    fullname: Strom
– volume-title: ISO 9241-210: Human-Centred Systems Design
  year: 2010
  ident: bibr4-1541931214581311
  contributor:
    fullname: International Organisation for Standardisation
– ident: bibr2-1541931214581311
  doi: 10.1001/archinternmed.2008.542
– volume: 27
  start-page: 509
  issue: 10
  year: 2001
  ident: bibr10-1541931214581311
  publication-title: Joint Commission Journal on Quality and Patient Safety
  contributor:
    fullname: Kuperman G. J.
– ident: bibr13-1541931214581311
  doi: 10.1518/001872006776412234
– volume-title: NISTIR 7742: Common Industry Format
  year: 2012
  ident: bibr16-1541931214581311
  contributor:
    fullname: Schumacher R
– volume-title: Handbook of Human Factors in Medical Device Design.
  year: 2011
  ident: bibr5-1541931214581311
  contributor:
    fullname: Israelski E
– volume: 13
  start-page: 547
  year: 2006
  ident: bibr3-1541931214581311
  publication-title: JAMIA
  contributor:
    fullname: Campbell EM
– volume: 17
  start-page: 617
  year: 2010
  ident: bibr7-1541931214581311
  publication-title: JAMIA
  contributor:
    fullname: Karsh BT
– volume: 15
  start-page: 408
  year: 2008
  ident: bibr9-1541931214581311
  publication-title: JAMIA
  contributor:
    fullname: Koppel R
– ident: bibr14-1541931214581311
  doi: 10.1177/154193120705101119
– ident: bibr1-1541931214581311
  doi: 10.1001/jama.1997.03540280045032
– volume-title: Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests
  year: 2008
  ident: bibr15-1541931214581311
  contributor:
    fullname: Rubin J
– volume-title: User Interface Design and Evaluation
  year: 2005
  ident: bibr17-1541931214581311
  contributor:
    fullname: Stone D
– ident: bibr12-1541931214581311
  doi: 10.1001/archinte.165.10.1111
– ident: bibr11-1541931214581311
  doi: 10.1001/jama.1995.03530010049034
– ident: bibr6-1541931214581311
  doi: 10.1001/jama.2010.1497
– ident: bibr8-1541931214581311
  doi: 10.1001/jama.293.10.1197
SSID ssj0023734
Score 1.9332215
Snippet The Office of the National Coordinator for Health Information Technology (ONC) has been tasked with managing the integration of usability best practices into...
SourceID crossref
sage
SourceType Aggregation Database
Publisher
StartPage 1491
Title High variability in Summative Usability Test methods & reporting among clinical informatics vendors complying with Federal certification requirements
URI https://journals.sagepub.com/doi/full/10.1177/1541931214581311
Volume 58
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1Na9wwEBVJ9tIeSj9p-oUOJVCC05Vl2fIxLVlCYUsKu5CeFkkrt724JfYW2v_R_9sZSZaV7RbSXswihMx6HqPR6M0bQl5WxjBlhM1Kk6Oodl1lNbjArJCGa8mNyl2h8Px9eb4s3l2Ky739ScJa2vT6xPzcWVfyP1aFMbArVsn-g2XjojAAv8G-8AQLw_NGNkaSxvF3OO16sW1XwodpLS_mveyG4QW4_tArunO29jcFrj7RNRuK9ZFBR9VpN4MbXGMrHsc6_xGTtjOUn0BVEWRkNyHlBysipdjlGrs04L2IG2Q30BH8vcEsNPrBxP3Z1SdfHt1FFmnQ_Z9bV5IdkwYbLFRW7SiLkNACFp8xAO4TSYTx2sklL31tkutvfvzmJEItVkPCO22aBWFFpHml1287E50QI7Ks5oGsbd1Yzso6E1PfCGTYB4T8A-_eqcMhkiUBAp4pd28-7vob34evY4WQqGU0brSR_hgmr7an7pNJDv4SHPXk9OPFh3nMHPDK0yOGvzLet7_eXuNafJWQE128tLhL7oSDDj31qL1H9mx7n9xO5C8fkF-IX5rgl35pacQvjfiliF8a8EuPaEQvdeilA3ppgl4a0Esjeimilwb00mvopSl6H5Ll7Gzx9jwLbUIyw-q6z0wuVCO4mgppbCEtb3QtC73msjHrsjRsrWohmrW0WjLWiKJRsqqkLVVZ6SYX_BE5aL-29jGhXJZKawjZrTUFq6bKVLBDwaE9103NG3NIXg0fd_XNq8GsWBDM3zbEITnCr78K_qL768QnN534lNwaof-MHPRXG_scQuJevwiA-Q3YyLnA
link.rule.ids 315,783,787,27936,27937
linkProvider SAGE Publications
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=High+variability+in+Summative+Usability+Test+methods+%26+reporting+among+clinical+informatics+vendors+complying+with+Federal+certification+requirements&rft.jtitle=Proceedings+of+the+Human+Factors+and+Ergonomics+Society+Annual+Meeting&rft.au=Buchanan%2C+Christine&rft.au=Threatt%2C+Anthony&rft.au=Weinger%2C+Matthew+B.&rft.au=Miller%2C+Anne&rft.date=2014-09-01&rft.pub=SAGE+Publications&rft.issn=1541-9312&rft.eissn=2169-5067&rft.volume=58&rft.issue=1&rft.spage=1491&rft.epage=1495&rft_id=info:doi/10.1177%2F1541931214581311&rft.externalDocID=10.1177_1541931214581311
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1541-9312&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1541-9312&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1541-9312&client=summon