Ensuring machine learning for healthcare works for all

• Published in: BMJ health & care informatics Vol. 27; no. 3; p. e100237
• Main Authors: McCoy, Liam G, Banja, John D, Ghassemi, Marzyeh, Celi, Leo Anthony
• Format: Journal Article
• Language: English
• Published: England BMJ Publishing Group Ltd 24.11.2020; BMJ Publishing Group LTD; BMJ Publishing Group
• Subjects: Artificial intelligence; Bias; Big Data; Clinical practice guidelines; Clinical trials; Commentary; Delivery of Health Care - methods; Delivery of Health Care - standards; health care; Health disparities; Health informatics; Humans; Information technology; Machine learning; Machine Learning - standards; medical informatics; patient care; Patients; patient care; health care; medical informatics
• ISSN: 2632-1009; 2632-1009
• DOI: 10.1136/bmjhci-2020-100237

Even when prospective randomised controlled trials are performed, they are subject to numerous opportunities for bias—and even outright conflict of interest—which can impact the quality and transferability of results.15 16 The burdens of medicine’s failures in evidentiary quality and applicability are not borne equally.11 17–19 The historical and ongoing omission in research of certain groups, including women and underserved populations, has skewed our understanding of health and disease.11 The concerns that exist regarding the generation of algorithms on racially biased datasets17 are unfortunately far from being new, but represent a continuation of a long-standing history of minority groups being under-represented or entirely unrepresented in foundational clinical research.11 18 The Framingham study, for example, generated its cardiovascular risk scores from an overwhelmingly white and male population, and has subsequently been inaccurate when uncritically used on black populations.19 Similarly, women have been and continue to be heavily under-represented in clinical trials.11 20 21 These problems extend to the global health context as well, as the trials used to inform clinical practice guidelines around the world tend to be conducted on a demographically restricted group of patients in high-income countries (mainly white males in the USA)11 These issues are compounded by structural biases in medical education,22 and the biases of the healthcare providers tasked with interpreting and implementing this medical knowledge in the clinical context.23 Can MLHC help, or will it harm? Models that are trained uncritically on databases embedded with societal biases and disparities will end up learning, amplifying and propagating those biases and disparities under the guise of algorithmic pseudo-objectivity.2 17 24 25 Similarly, gaps in quality of care will be widened by the development and use of tools that are only beneficial to a certain population—such as a melanoma detection algorithm trained on a dataset containing mostly images of light toned skin.26 Concerns also exist around patient privacy and safeguarding sensitive data (particularly for vulnerable groups such as HIV positive patients).27 Finally, there are structural concerns related to the possibility that the information technology prerequisites for implementing MLHC will only be available to already privileged groups.5 7 Yet, and as recent scholarship has indicated, the potential for MLHC to counter biases in healthcare is considerable.3 28 Data science methods can be used to audit healthcare datasets and processes, deriving insights and exposing implicit biases so they might be directly investigated and addressed.1 3 29 While much has been made of the ‘black box’ characteristics of AI, it may be argued that human decision making in general is no more explainable.30 31 This is particularly true in the context of the sort of implicit gender and racial biases that influence physicians' decisions but are unlikely to be consciously admitted.23 As checklist studies in healthcare have demonstrated,32 it may be possible to reduce these biases through the use of standardised prompts and clinical decision support tools that move clinical decisions closer to the data—and further from the biasing subjective evaluations. At the structural level, there is hope that AI will drive down the costs of care, increasing access for groups that have been traditionally underserved, and enabling greater levels of patient autonomy for self-management.4 5 Further, MLHC technologies may be able to address issues of disparity in the clinical research pipeline.33 Improvements in the use and analysis of electronic health records and mobile health technology herald the possibility of mobilising massive amounts of healthcare data from across domestic and global populations. Embracing it must not lead subsequently to the neglect of the role played by other structural factors such as economic inequities36 and implicit physician bias.23 No simple set of data-focused technical interventions alone can effectively deal with complex sociopolitical environments and structural inequity,37 and simple ‘race correction’ methods can be deeply problematic.38 The potential for ‘big data’ synthetic clinical trials, for example, must come as a supplement to and not a replacement for efforts to improve the diversity of clinical trial recruitment.