Human-AI Teaming in Critical Care: A Comparative Analysis of Data Scientists' and Clinicians' Perspectives on AI Augmentation and Automation

• Published in: Journal of medical Internet research Vol. 26; no. 7956; p. e50130
• Main Authors: Bienefeld, Nadine, Keller, Emanuela, Grote, Gudela
• Format: Journal Article
• Language: English
• Published: Canada Journal of Medical Internet Research 22.07.2024; JMIR Publications
• Subjects: Artificial Intelligence; Automation; Care and treatment; Critical Care - methods; Data Science - methods; Delphi Technique; Ethical aspects; Female; Health care reform; Hospital patients; Humans; Intensive Care Units; Male; Nurses; Original Paper; Physician and patient; Scientists; Social networks; policies; policy maker; recommendation; explainable AI; ethical implications of AI in health care; AI adoption; augmentation; ethics; recommendations; artificial intelligence; AI implementation; human-computer; policy; intensive care; human-AI teaming; sociotechnical systems; human-machine; ethical; Delphi; explainable; foresight; sociotechnical; human-AI; ethic; policy makers; AI in health care; ICU; automation, health care policy and regulatory foresight
• ISSN: 1438-8871; 1439-4456; 1438-8871
• DOI: 10.2196/50130

Artificial intelligence (AI) holds immense potential for enhancing clinical and administrative health care tasks. However, slow adoption and implementation challenges highlight the need to consider how humans can effectively collaborate with AI within broader socio-technical systems in health care. In the example of intensive care units (ICUs), we compare data scientists' and clinicians' assessments of the optimal utilization of human and AI capabilities by determining suitable levels of human-AI teaming for safely and meaningfully augmenting or automating 6 core tasks. The goal is to provide actionable recommendations for policy makers and health care practitioners regarding AI design and implementation. In this multimethod study, we combine a systematic task analysis across 6 ICUs with an international Delphi survey involving 19 health data scientists from the industry and academia and 61 ICU clinicians (25 physicians and 36 nurses) to define and assess optimal levels of human-AI teaming (level 1=no performance benefits; level 2=AI augments human performance; level 3=humans augment AI performance; level 4=AI performs without human input). Stakeholder groups also considered ethical and social implications. Both stakeholder groups chose level 2 and 3 human-AI teaming for 4 out of 6 core tasks in the ICU. For one task (monitoring), level 4 was the preferred design choice. For the task of patient interactions, both data scientists and clinicians agreed that AI should not be used regardless of technological feasibility due to the importance of the physician-patient and nurse-patient relationship and ethical concerns. Human-AI design choices rely on interpretability, predictability, and control over AI systems. If these conditions are not met and AI performs below human-level reliability, a reduction to level 1 or shifting accountability away from human end users is advised. If AI performs at or beyond human-level reliability and these conditions are not met, shifting to level 4 automation should be considered to ensure safe and efficient human-AI teaming. By considering the sociotechnical system and determining appropriate levels of human-AI teaming, our study showcases the potential for improving the safety and effectiveness of AI usage in ICUs and broader health care settings. Regulatory measures should prioritize interpretability, predictability, and control if clinicians hold full accountability. Ethical and social implications must be carefully evaluated to ensure effective collaboration between humans and AI, particularly considering the most recent advancements in generative AI.