Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation

• Published in: Journal of biomedical informatics Vol. 127; p. 104014
• Main Authors: Kukhareva, Polina V., Weir, Charlene, Del Fiol, Guilherme, Aarons, Gregory A., Taft, Teresa Y., Schlechter, Chelsey R., Reese, Thomas J., Curran, Rebecca L., Nanjo, Claude, Borbolla, Damian, Staes, Catherine J., Morgan, Keaton L., Kramer, Heidi S., Stipelman, Carole H., Shakib, Julie H., Flynn, Michael C., Kawamoto, Kensaku
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.03.2022
• Subjects: Clinical decision support; Commerce; Electronic Health Records; Evaluation framework; Health information technology; Health technology assessment; Human factors engineering; Humans; Implementation science; Information Technology; Medical Informatics; Technology; Evaluation framework; Human factors engineering; Health technology assessment; Clinical decision support; Implementation science; Health information technology
• DOI: 10.1016/j.jbi.2022.104014
• ISSN: 1532-0464

[Display omitted] •It is critical for EHR-integrated innovations to be systematically evaluated.•Effective evaluation of EHR-integrated innovations requires a shared understanding and collaboration across disciplines throughout the full information technology (IT) life cycle.•A novel Evaluation in Life Cycle of Information Technology (ELICIT) framework focuses on all phases of the IT life cycle.•The ELICIT framework consists of 12 evaluation steps across the IT life cycle phases including society-, user-, and IT-level measures. Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.