Automating hybrid collective intelligence in open-ended medical diagnostics

• Published in: Proceedings of the National Academy of Sciences - PNAS Vol. 120; no. 34; p. e2221473120
• Main Authors: Kurvers, Ralf H J M, Nuzzolese, Andrea Giovanni, Russo, Alessandro, Barabucci, Gioele, Herzog, Stefan M, Trianni, Vito
• Format: Journal Article
• Language: English
• Published: United States National Academy of Sciences 22.08.2023
• Subjects: Accuracy; Automation; Biological Sciences; Decision making; Diagnosis; Emergency management; Emergency medical services; Emergency preparedness; Intelligence; Intelligence (information); Knowledge representation; Natural language processing; Social Sciences; Unstructured data; general medical diagnostics; natural language processing; collective intelligence; ontology
• ISSN: 0027-8424; 1091-6490
• DOI: 10.1073/pnas.2221473120

Collective intelligence has emerged as a powerful mechanism to boost decision accuracy across many domains, such as geopolitical forecasting, investment, and medical diagnostics. However, collective intelligence has been mostly applied to relatively simple decision tasks (e.g., binary classifications). Applications in more open-ended tasks with a much larger problem space, such as emergency management or general medical diagnostics, are largely lacking, due to the challenge of integrating unstandardized inputs from different crowd members. Here, we present a fully automated approach for harnessing collective intelligence in the domain of general medical diagnostics. Our approach leverages semantic knowledge graphs, natural language processing, and the SNOMED CT medical ontology to overcome a major hurdle to collective intelligence in open-ended medical diagnostics, namely to identify the intended diagnosis from unstructured text. We tested our method on 1,333 medical cases diagnosed on a medical crowdsourcing platform: The Human Diagnosis Project. Each case was independently rated by ten diagnosticians. Comparing the diagnostic accuracy of single diagnosticians with the collective diagnosis of differently sized groups, we find that our method substantially increases diagnostic accuracy: While single diagnosticians achieved 46% accuracy, pooling the decisions of ten diagnosticians increased this to 76%. Improvements occurred across medical specialties, chief complaints, and diagnosticians' tenure levels. Our results show the life-saving potential of tapping into the collective intelligence of the global medical community to reduce diagnostic errors and increase patient safety.