Implementing clinical decision support for oncology advanced care planning: A systems engineering framework to optimize the usability and utility of a machine learning predictive model in clinical practice
Abstract only 330 Background: Machine learning models are well-positioned to transform cancer care delivery by providing oncologists with more accurate or accessible information to augment clinical decisions. Many machine learning projects, however, focus on model accuracy without considering the im...
Saved in:
Published in | Journal of clinical oncology Vol. 39; no. 28_suppl; p. 330 |
---|---|
Main Authors | , , , , , , , , , , , , |
Format | Journal Article |
Language | English |
Published |
01.10.2021
|
Online Access | Get full text |
Cover
Loading…
Abstract | Abstract only
330
Background: Machine learning models are well-positioned to transform cancer care delivery by providing oncologists with more accurate or accessible information to augment clinical decisions. Many machine learning projects, however, focus on model accuracy without considering the impact of using the model in real-world settings and rarely carry forward to clinical implementation. We present a human-centered systems engineering approach to address clinical problems with workflow interventions utilizing machine learning algorithms. Methods: We aimed to develop a mortality predictive tool, using a Random Forest algorithm, to identify oncology patients at high risk of death within 30 days to move advance care planning (ACP) discussions earlier in the illness trajectory. First, a project sponsor defined the clinical need and requirements of an intervention. The data scientists developed the predictive algorithm using data available in the electronic health record (EHR). A multidisciplinary workgroup was assembled including oncology physicians, advanced practice providers, nurses, social workers, chaplain, clinical informaticists, and data scientists. Meeting bi-monthly, the group utilized human-centered design (HCD) methods to understand clinical workflows and identify points of intervention. The workgroup completed a workflow redesign workshop, a 90-minute facilitated group discussion, to integrate the model in a future state workflow. An EHR (Epic) analyst built the user interface to support the intervention per the group’s requirements. The workflow was piloted in thoracic oncology and bone marrow transplant with plans to scale to other cancer clinics. Results: Our predictive model performance on test data was acceptable (sensitivity 75%, specificity 75%, F-1 score 0.71, AUC 0.82). The workgroup identified a “quality of life coordinator” who: reviews an EHR report of patients scheduled in the upcoming 7 days who have a high risk of 30-day mortality; works with the oncology team to determine ACP clinical appropriateness; documents the need for ACP; identifies potential referrals to supportive oncology, social work, or chaplain; and coordinates the oncology appointment. The oncologist receives a reminder on the day of the patient’s scheduled visit. Conclusions: This workgroup is a viable approach that can be replicated at institutions to address clinical needs and realize the full potential of machine learning models in healthcare. The next steps for this project are to address end-user feedback from the pilot, expand the intervention to other cancer disease groups, and track clinical metrics. |
---|---|
AbstractList | Abstract only
330
Background: Machine learning models are well-positioned to transform cancer care delivery by providing oncologists with more accurate or accessible information to augment clinical decisions. Many machine learning projects, however, focus on model accuracy without considering the impact of using the model in real-world settings and rarely carry forward to clinical implementation. We present a human-centered systems engineering approach to address clinical problems with workflow interventions utilizing machine learning algorithms. Methods: We aimed to develop a mortality predictive tool, using a Random Forest algorithm, to identify oncology patients at high risk of death within 30 days to move advance care planning (ACP) discussions earlier in the illness trajectory. First, a project sponsor defined the clinical need and requirements of an intervention. The data scientists developed the predictive algorithm using data available in the electronic health record (EHR). A multidisciplinary workgroup was assembled including oncology physicians, advanced practice providers, nurses, social workers, chaplain, clinical informaticists, and data scientists. Meeting bi-monthly, the group utilized human-centered design (HCD) methods to understand clinical workflows and identify points of intervention. The workgroup completed a workflow redesign workshop, a 90-minute facilitated group discussion, to integrate the model in a future state workflow. An EHR (Epic) analyst built the user interface to support the intervention per the group’s requirements. The workflow was piloted in thoracic oncology and bone marrow transplant with plans to scale to other cancer clinics. Results: Our predictive model performance on test data was acceptable (sensitivity 75%, specificity 75%, F-1 score 0.71, AUC 0.82). The workgroup identified a “quality of life coordinator” who: reviews an EHR report of patients scheduled in the upcoming 7 days who have a high risk of 30-day mortality; works with the oncology team to determine ACP clinical appropriateness; documents the need for ACP; identifies potential referrals to supportive oncology, social work, or chaplain; and coordinates the oncology appointment. The oncologist receives a reminder on the day of the patient’s scheduled visit. Conclusions: This workgroup is a viable approach that can be replicated at institutions to address clinical needs and realize the full potential of machine learning models in healthcare. The next steps for this project are to address end-user feedback from the pilot, expand the intervention to other cancer disease groups, and track clinical metrics. |
Author | Crow, Madalene Pappalardo, Rachel Freeman, Robbie Parchure, Prathamesh Bhardwaj, Aarti Sonia Ganta, Teja Kia, Arash Mazumdar, Madhu Will, Meagan Keyzner, Alla Smith, Cardinale B. Liu, Mark Lehrman, Stephanie |
Author_xml | – sequence: 1 givenname: Teja surname: Ganta fullname: Ganta, Teja organization: Tisch Cancer Institute, Division of Hematology and Medical Oncology, Icahn School of Medicine at Mount Sinai, New York, NY – sequence: 2 givenname: Stephanie surname: Lehrman fullname: Lehrman, Stephanie organization: Tisch Cancer Institute, Division of Hematology and Medical Oncology, Icahn School of Medicine at Mount Sinai, New York, NY – sequence: 3 givenname: Rachel surname: Pappalardo fullname: Pappalardo, Rachel organization: Tisch Cancer Institute, Division of Hematology and Medical Oncology, Icahn School of Medicine at Mount Sinai, New York, NY – sequence: 4 givenname: Madalene surname: Crow fullname: Crow, Madalene organization: Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY – sequence: 5 givenname: Meagan surname: Will fullname: Will, Meagan organization: Health System Operations, Mount Sinai Health System, New York, NY – sequence: 6 givenname: Mark surname: Liu fullname: Liu, Mark organization: Tisch Cancer Institute, Icahn School of Medicine at Mount Sinai, New York, NY – sequence: 7 givenname: Robbie surname: Freeman fullname: Freeman, Robbie organization: Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY – sequence: 8 givenname: Arash surname: Kia fullname: Kia, Arash organization: Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY – sequence: 9 givenname: Prathamesh surname: Parchure fullname: Parchure, Prathamesh organization: Institute for Healthcare Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY – sequence: 10 givenname: Alla surname: Keyzner fullname: Keyzner, Alla organization: Tisch Cancer Institute, Division of Hematology and Medical Oncology, Icahn School of Medicine at Mount Sinai, New York, NY – sequence: 11 givenname: Madhu surname: Mazumdar fullname: Mazumdar, Madhu organization: Institute for Healthcare Delivery Science, Tisch Cancer Institute, Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, NY – sequence: 12 givenname: Aarti Sonia surname: Bhardwaj fullname: Bhardwaj, Aarti Sonia organization: Tisch Cancer Institute, Division of Hematology and Medical Oncology, Icahn School of Medicine at Mount Sinai, New York, NY – sequence: 13 givenname: Cardinale B. surname: Smith fullname: Smith, Cardinale B. organization: Division of Hematology and Medical Oncology, Tisch Cancer Institute at Mount Sinai, New York, NY |
BookMark | eNqlkE1OwzAQhS1UJFLgDnOBBCcmSsoOVSBgw4YFO8s4k3bAf7LdonJH7kQiQByA1Txp9D49fUu2cN4hY1Dzqm44v3hYP1YNb3glVlXTy7QLwVRC8CNW1G3TlV3XtgtW8E40Zd2L5xO2TOmV8_qyF23BPu9tMGjRZXIb0IYcaWVgQE2JvIOZ52OG0UfwTnvjNwdQw145jQNoFRGCUc5N7Su4hnRIGW0CdBtyiHGGjlFZfPfxDbIHHzJZ-kDIW4RdUi9kKE9EN8Auf2c_ggKr9HYigEEVZziEiAPpTHsE6wc0QO5vbohqemk8Y8ejMgnPf-4p629vntZ3pY4-pYijDJGsigdZcznrk5M-OeuTYiV_9clJn_hH9QvgZIcy |
ContentType | Journal Article |
DBID | AAYXX CITATION |
DOI | 10.1200/JCO.2020.39.28_suppl.330 |
DatabaseName | CrossRef |
DatabaseTitle | CrossRef |
DatabaseTitleList | CrossRef |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Medicine Pharmacy, Therapeutics, & Pharmacology |
EISSN | 1527-7755 |
EndPage | 330 |
ExternalDocumentID | 10_1200_JCO_2020_39_28_suppl_330 |
GroupedDBID | --- .55 0R~ 18M 2WC 34G 39C 4.4 53G 5GY 5RE 8F7 AAQQT AARDX AAWTL AAYEP AAYXX ABJNI ABOCM ACGFO ACGFS ACGUR ADBBV AEGXH AENEX AIAGR ALMA_UNASSIGNED_HOLDINGS AWKKM BAWUL C45 CITATION CS3 DIK EBS EJD F5P F9R FBNNL FD8 GX1 H13 HZ~ IH2 IPNFZ K-O KQ8 L7B LSO MJL N9A O9- OK1 OVD OWW P2P QTD R1G RHI RIG RLZ RUC SJN SV3 TEORI TR2 TWZ UDS VVN WH7 X7M YCJ YFH YQY |
ID | FETCH-crossref_primary_10_1200_JCO_2020_39_28_suppl_3303 |
ISSN | 0732-183X |
IngestDate | Fri Aug 23 01:53:11 EDT 2024 |
IsPeerReviewed | true |
IsScholarly | true |
Issue | 28_suppl |
Language | English |
LinkModel | OpenURL |
MergedId | FETCHMERGED-crossref_primary_10_1200_JCO_2020_39_28_suppl_3303 |
ParticipantIDs | crossref_primary_10_1200_JCO_2020_39_28_suppl_330 |
PublicationCentury | 2000 |
PublicationDate | 2021-10-01 |
PublicationDateYYYYMMDD | 2021-10-01 |
PublicationDate_xml | – month: 10 year: 2021 text: 2021-10-01 day: 01 |
PublicationDecade | 2020 |
PublicationTitle | Journal of clinical oncology |
PublicationYear | 2021 |
SSID | ssj0014835 |
Score | 4.8450384 |
Snippet | Abstract only
330
Background: Machine learning models are well-positioned to transform cancer care delivery by providing oncologists with more accurate or... |
SourceID | crossref |
SourceType | Aggregation Database |
StartPage | 330 |
Title | Implementing clinical decision support for oncology advanced care planning: A systems engineering framework to optimize the usability and utility of a machine learning predictive model in clinical practice |
Volume | 39 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1Nb9NAEF21RUJcEBQQ35oD6iV1aNaOHXOrIqBClFYoSLlZTrwWRa1tNc6h_Y_9T53ZnVlbQAXtxUosZ7Tredmd3X3zRql3id5blPEiD1JT5kFUxGmAT46DUZyXtEWok9KyLb7FBz-iL_PxfGMTeqyldbsYLi__mldyF6_iPfQrZcnewrPeKN7Az-hfvKKH8fpfPrbSvpbvQ5mzkuNYcNmcwWrdUHRtmYR1tWS1JTn0t5yvhmsWuQR1J-u8GphOpHBQCnuLotQaB5izk0tjw9W1U-dtnYITdvOU6R354MxSNI3UpKB8dzoQsjQlW3uHtll8gyVT64ZA2T8nffCcIUSFjX1n5pefXb6an-eyq0sMNkqh787JmibHpXxh94e_k5a1J5hMWYvyMCeKNZMNeDtEjzyxTkbNJNQBjlNzN8HxqK4TXEY4PWAZ9p2GEsNbTzLyyWlvKA_5vMj0v_0x4WhXS3t6NMTG7A0p94lNDb2Bvsb3b3OvZ0TSWkzTweL0KCNLWZhmYilDS5vqnk7SMZFWP889iQnXsq6ErHSauWpo6f1NbeoFYL1IavZIPWTPwr7D82O1Yaptdf-QSR7baufYyalf7MKsyw5c7cIOHHdC6xdP1FUf_yAwAcE_MP4B8Q-CHRD8A-EfBP8fYB8Y_dBDP3j0Q1uDoB8Q_eDRD4h-YPRDXUIOjH4Q9EOHfrDoh5Oqa66g_6mafPo4mx4E8t6yxinBZP_yWvhMbVV1ZZ4rKOIl6S7FeRFNojChhdXS6DBMTbTUcaFfqNGtzb-8w29eqQfd3-a12mrP1-YNxtXt4q2F1jU1Kd-P |
link.rule.ids | 315,786,790,27955,27956 |
linkProvider | Geneva Foundation for Medical Education and Research |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Implementing+clinical+decision+support+for+oncology+advanced+care+planning%3A+A+systems+engineering+framework+to+optimize+the+usability+and+utility+of+a+machine+learning+predictive+model+in+clinical+practice&rft.jtitle=Journal+of+clinical+oncology&rft.au=Ganta%2C+Teja&rft.au=Lehrman%2C+Stephanie&rft.au=Pappalardo%2C+Rachel&rft.au=Crow%2C+Madalene&rft.date=2021-10-01&rft.issn=0732-183X&rft.eissn=1527-7755&rft.volume=39&rft.issue=28_suppl&rft.spage=330&rft.epage=330&rft_id=info:doi/10.1200%2FJCO.2020.39.28_suppl.330&rft.externalDBID=n%2Fa&rft.externalDocID=10_1200_JCO_2020_39_28_suppl_330 |
thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0732-183X&client=summon |
thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0732-183X&client=summon |
thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0732-183X&client=summon |