ChatGPT performance on the American Shoulder and Elbow Surgeons maintenance of certification exam
While multiple studies have tested the ability of large language models (LLMs), such as ChatGPT, to pass standardized medical exams at different levels of training, LLMs have never been tested on surgical sub-specialty examinations, such as the American Shoulder and Elbow Surgeons (ASES) Maintenance...
Saved in:
Published in | Journal of shoulder and elbow surgery Vol. 33; no. 9; pp. 1888 - 1893 |
---|---|
Main Authors | , , , |
Format | Journal Article |
Language | English |
Published |
United States
Elsevier Inc
01.09.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | While multiple studies have tested the ability of large language models (LLMs), such as ChatGPT, to pass standardized medical exams at different levels of training, LLMs have never been tested on surgical sub-specialty examinations, such as the American Shoulder and Elbow Surgeons (ASES) Maintenance of Certification (MOC). The purpose of this study was to compare results of ChatGPT 3.5, GPT-4, and fellowship-trained surgeons on the 2023 ASES MOC self-assessment exam.
ChatGPT 3.5 and GPT-4 were subjected to the same set of text-only questions from the ASES MOC exam, and GPT-4 was additionally subjected to image-based MOC exam questions. Question responses from both models were compared against the correct answers. Performance of both models was compared to corresponding average human performance on the same question subsets. One sided proportional z-test were utilized to analyze data.
Humans performed significantly better than Chat GPT 3.5 on exclusively text-based questions (76.4% vs. 60.8%, P = .044). Humans also performed significantly better than GPT 4 on image-based questions (73.9% vs. 53.2%, P = .019). There was no significant difference between humans and GPT 4 in text-based questions (76.4% vs. 66.7%, P = .136). Accounting for all questions, humans significantly outperformed GPT-4 (75.3% vs. 60.2%, P = .012). GPT-4 did not perform statistically significantly betterer than ChatGPT 3.5 on text-only questions (66.7% vs. 60.8%, P = .268).
Although human performance was overall superior, ChatGPT demonstrated the capacity to analyze orthopedic information and answer specialty-specific questions on the ASES MOC exam for both text and image-based questions. With continued advancements in deep learning, LLMs may someday rival exam performance of fellowship-trained surgeons. |
---|---|
AbstractList | While multiple studies have tested the ability of large language models (LLM), such as ChatGPT, to pass standardized medical exams at different levels of training, LLMs have never been tested on surgical sub-specialty examinations, such as the American Shoulder and Elbow Surgeons (ASES) Maintenance of Certification (MOC). The purpose of this study was to compare results of ChatGPT 3.5, GPT-4, and fellowship-trained surgeons on the 2023 American Shoulder and Elbow Surgeons (ASES) Maintenance of Certification (MOC) self-assessment exam.
ChatGPT 3.5 and GPT-4 were subjected to the same set of text-only questions from the ASES MOC exam, and GPT-4 was additionally subjected to image-based MOC exam questions. Question responses from both models were compared against the correct answers. Performance of both models was compared to corresponding average human performance on the same question subsets. One sided proportional z-test were utilized to analyze data.
Humans performed significantly better than Chat GPT 3.5 on exclusively text-based questions (76.4% vs. 60.8%, p= .044). Humans also performed significantly better than GPT 4 on image-based questions (73.9% vs. 53.2%, p= .019). There was no significant difference between humans and GPT 4 in text-based questions (76.4% vs. 66.7%, p=0.136). Accounting for all questions, humans significantly outperformed GPT-4 (75.3% vs. 60.2%, p= .012). GPT-4 did not perform statistically significantly betterer than ChatGPT 3.5 on text-only questions (66.7% vs. 60.8%, p= .268).
Although human performance was overall superior, ChatGPT demonstrated the capacity to analyze orthopedic information and answer specialty-specific questions on the ASES MOC exam for both text and image-based questions. With continued advancements in deep learning, large language models may someday rival exam performance of fellowship-trained surgeons. While multiple studies have tested the ability of large language models (LLMs), such as ChatGPT, to pass standardized medical exams at different levels of training, LLMs have never been tested on surgical sub-specialty examinations, such as the American Shoulder and Elbow Surgeons (ASES) Maintenance of Certification (MOC). The purpose of this study was to compare results of ChatGPT 3.5, GPT-4, and fellowship-trained surgeons on the 2023 ASES MOC self-assessment exam.BACKGROUNDWhile multiple studies have tested the ability of large language models (LLMs), such as ChatGPT, to pass standardized medical exams at different levels of training, LLMs have never been tested on surgical sub-specialty examinations, such as the American Shoulder and Elbow Surgeons (ASES) Maintenance of Certification (MOC). The purpose of this study was to compare results of ChatGPT 3.5, GPT-4, and fellowship-trained surgeons on the 2023 ASES MOC self-assessment exam.ChatGPT 3.5 and GPT-4 were subjected to the same set of text-only questions from the ASES MOC exam, and GPT-4 was additionally subjected to image-based MOC exam questions. Question responses from both models were compared against the correct answers. Performance of both models was compared to corresponding average human performance on the same question subsets. One sided proportional z-test were utilized to analyze data.METHODSChatGPT 3.5 and GPT-4 were subjected to the same set of text-only questions from the ASES MOC exam, and GPT-4 was additionally subjected to image-based MOC exam questions. Question responses from both models were compared against the correct answers. Performance of both models was compared to corresponding average human performance on the same question subsets. One sided proportional z-test were utilized to analyze data.Humans performed significantly better than Chat GPT 3.5 on exclusively text-based questions (76.4% vs. 60.8%, P = .044). Humans also performed significantly better than GPT 4 on image-based questions (73.9% vs. 53.2%, P = .019). There was no significant difference between humans and GPT 4 in text-based questions (76.4% vs. 66.7%, P = .136). Accounting for all questions, humans significantly outperformed GPT-4 (75.3% vs. 60.2%, P = .012). GPT-4 did not perform statistically significantly betterer than ChatGPT 3.5 on text-only questions (66.7% vs. 60.8%, P = .268).RESULTSHumans performed significantly better than Chat GPT 3.5 on exclusively text-based questions (76.4% vs. 60.8%, P = .044). Humans also performed significantly better than GPT 4 on image-based questions (73.9% vs. 53.2%, P = .019). There was no significant difference between humans and GPT 4 in text-based questions (76.4% vs. 66.7%, P = .136). Accounting for all questions, humans significantly outperformed GPT-4 (75.3% vs. 60.2%, P = .012). GPT-4 did not perform statistically significantly betterer than ChatGPT 3.5 on text-only questions (66.7% vs. 60.8%, P = .268).Although human performance was overall superior, ChatGPT demonstrated the capacity to analyze orthopedic information and answer specialty-specific questions on the ASES MOC exam for both text and image-based questions. With continued advancements in deep learning, LLMs may someday rival exam performance of fellowship-trained surgeons.DISCUSSIONAlthough human performance was overall superior, ChatGPT demonstrated the capacity to analyze orthopedic information and answer specialty-specific questions on the ASES MOC exam for both text and image-based questions. With continued advancements in deep learning, LLMs may someday rival exam performance of fellowship-trained surgeons. While multiple studies have tested the ability of large language models (LLMs), such as ChatGPT, to pass standardized medical exams at different levels of training, LLMs have never been tested on surgical sub-specialty examinations, such as the American Shoulder and Elbow Surgeons (ASES) Maintenance of Certification (MOC). The purpose of this study was to compare results of ChatGPT 3.5, GPT-4, and fellowship-trained surgeons on the 2023 ASES MOC self-assessment exam. ChatGPT 3.5 and GPT-4 were subjected to the same set of text-only questions from the ASES MOC exam, and GPT-4 was additionally subjected to image-based MOC exam questions. Question responses from both models were compared against the correct answers. Performance of both models was compared to corresponding average human performance on the same question subsets. One sided proportional z-test were utilized to analyze data. Humans performed significantly better than Chat GPT 3.5 on exclusively text-based questions (76.4% vs. 60.8%, P = .044). Humans also performed significantly better than GPT 4 on image-based questions (73.9% vs. 53.2%, P = .019). There was no significant difference between humans and GPT 4 in text-based questions (76.4% vs. 66.7%, P = .136). Accounting for all questions, humans significantly outperformed GPT-4 (75.3% vs. 60.2%, P = .012). GPT-4 did not perform statistically significantly betterer than ChatGPT 3.5 on text-only questions (66.7% vs. 60.8%, P = .268). Although human performance was overall superior, ChatGPT demonstrated the capacity to analyze orthopedic information and answer specialty-specific questions on the ASES MOC exam for both text and image-based questions. With continued advancements in deep learning, LLMs may someday rival exam performance of fellowship-trained surgeons. |
Author | Azua, Eric N. Fiedler, Benjamin Phillips, Todd Ahmed, Adil Shahzad |
Author_xml | – sequence: 1 givenname: Benjamin orcidid: 0000-0001-7484-2348 surname: Fiedler fullname: Fiedler, Benjamin email: Benjamin.Fiedler@bcm.edu – sequence: 2 givenname: Eric N. orcidid: 0000-0001-7531-0175 surname: Azua fullname: Azua, Eric N. – sequence: 3 givenname: Todd orcidid: 0000-0001-7382-5191 surname: Phillips fullname: Phillips, Todd – sequence: 4 givenname: Adil Shahzad orcidid: 0000-0002-7064-8430 surname: Ahmed fullname: Ahmed, Adil Shahzad |
BackLink | https://www.ncbi.nlm.nih.gov/pubmed/38580067$$D View this record in MEDLINE/PubMed |
BookMark | eNp9kE1r3DAQhkVJaT5_QC5Fx1y8GUuW1yKnsCRpIdDCbs9ClkZZLba0lew2-fdR2LTHwMDM4XlfmOeUHIUYkJDLGhY11O31brHLuGDAmgWwMvITOakFZ1UrAI7KDaKr2LJpj8lpzjsAkA2wL-SYd6IDaJcnRK-2enr4uaF7TC6mUQeDNAY6bZHejpi80YGut3EeLCaqg6V3Qx__0vWcnjCGTEftw4ThkHPUYJq8K6nJlxZ81uM5-ez0kPHifZ-RX_d3m9W36vHHw_fV7WNlGBdTJWzvWK9bLQQuObdLzZExLZ0UVqDrTFNrwQV3rHZCSg0SoXfSuo5JA4zxM3J16N2n-HvGPKnRZ4PDoAPGOSsOvGFNx7umoPUBNSnmnNCpffKjTi-qBvVmVu1UMavezCpgZWTJfH2vn_sR7f_EP5UFuDkAWJ784zGpbDwWLdYnNJOy0X9Q_wowT4uU |
Cites_doi | 10.1080/17453674.2018.1453714 10.1016/j.jse.2023.03.028 10.1002/jmri.25750 10.1097/CORR.0000000000002668 10.3399/bjgpopen17X100833 10.1007/s00330-022-09349-3 10.4103/0301-4738.156934 10.1038/s41746-020-00352-w 10.3389/frai.2023.1169595 10.1007/s00256-022-04008-6 10.1097/CORR.0000000000002704 10.2196/45312 10.1371/journal.pdig.0000198 10.1016/j.jseint.2023.07.018 10.1002/jor.23519 10.30953/bhty.v6.273 10.1126/science.1891713 10.1093/fampra/cmab099 10.1177/0363546519892905 10.1007/s10439-023-03272-4 10.1007/s12178-020-09600-8 10.1177/00031348221101503 10.1227/neu.0000000000002632 10.1136/postgradmedj-2022-141596 10.1016/j.cmpb.2016.12.006 |
ContentType | Journal Article |
Copyright | 2024 Journal of Shoulder and Elbow Surgery Board of Trustees Copyright © 2024. Published by Elsevier Inc. Copyright © 2024 Journal of Shoulder and Elbow Surgery Board of Trustees. Published by Elsevier Inc. All rights reserved. |
Copyright_xml | – notice: 2024 Journal of Shoulder and Elbow Surgery Board of Trustees – notice: Copyright © 2024. Published by Elsevier Inc. – notice: Copyright © 2024 Journal of Shoulder and Elbow Surgery Board of Trustees. Published by Elsevier Inc. All rights reserved. |
DBID | NPM AAYXX CITATION 7X8 |
DOI | 10.1016/j.jse.2024.02.029 |
DatabaseName | PubMed CrossRef MEDLINE - Academic |
DatabaseTitle | PubMed CrossRef MEDLINE - Academic |
DatabaseTitleList | PubMed MEDLINE - Academic |
Database_xml | – sequence: 1 dbid: NPM name: PubMed url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database |
DeliveryMethod | fulltext_linktorsrc |
EISSN | 1532-6500 |
EndPage | 1893 |
ExternalDocumentID | 10_1016_j_jse_2024_02_029 38580067 S1058274624002313 |
Genre | Journal Article |
GroupedDBID | --- --K --M .1- .FO .~1 0R~ 1B1 1P~ 1RT 1~. 1~5 4.4 457 4G. 53G 5GY 5VS 7-5 71M 8P~ 9JM AACTN AAEDT AAEDW AAIKJ AAKAS AAKOC AALRI AAOAW AAQFI AAQQT AAQXK AAXUO ABBQC ABFNM ABJNI ABMAC ABMZM ABXDB ACDAQ ACGFS ACRLP ADBBV ADEZE ADMUD ADZCM AEBSH AEKER AENEX AEVXI AFCTW AFKWA AFRHN AFTJW AFXIZ AGHFR AGUBO AGYEJ AHHHB AIEXJ AIKHN AITUG AJOXV AJRQY AJUYK AKRWK ALMA_UNASSIGNED_HOLDINGS AMFUW AMRAJ ANZVX ASPBG AVWKF AXJTR AZFZN BKOJK BLXMC BNPGV CAG COF CS3 DU5 EBS EFJIC EJD EO8 EO9 EP2 EP3 FDB FEDTE FGOYB FIRID FNPLU FYGXN G-2 G-Q GBLVA HEK HMK HMO HVGLF HZ~ IHE J1W KOM M41 MO0 N9A O-L O9- OAUVE OF0 OR. OVD OZT P-8 P-9 P2P PC. Q38 R2- RIG ROL RPZ SAE SCC SDF SDG SEL SES SEW SPCBC SSH SSZ T5K TEORI UHS UV1 WUQ YFH Z5R ~G- AAXKI AFJKZ NPM AAYXX ACRPL ADNMO CITATION 7X8 |
ID | FETCH-LOGICAL-c235t-5dbf2ba6a55e733d7a3e22a9f95d5ef8c41a5353f21f599a09e0bf9df829c0223 |
IEDL.DBID | .~1 |
ISSN | 1058-2746 1532-6500 |
IngestDate | Thu Dec 05 22:41:26 EST 2024 Fri Dec 06 09:16:32 EST 2024 Sat Nov 02 12:16:04 EDT 2024 Sat Aug 17 15:41:27 EDT 2024 |
IsPeerReviewed | true |
IsScholarly | true |
Issue | 9 |
Keywords | elbow ChatGPT artificial intelligence (AI) maintenance of certification (MOC) Basic Science Study Validation of Computer Modeling machine learning shoulder American Shoulder and Elbow Surgeons (ASES) Artificial Intelligence (AI) Shoulder Maintenance of Certification (MOC) Elbow Machine Learning |
Language | English |
License | Copyright © 2024. Published by Elsevier Inc. |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-c235t-5dbf2ba6a55e733d7a3e22a9f95d5ef8c41a5353f21f599a09e0bf9df829c0223 |
Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
ORCID | 0000-0001-7531-0175 0000-0001-7382-5191 0000-0002-7064-8430 0000-0001-7484-2348 |
PMID | 38580067 |
PQID | 3034248384 |
PQPubID | 23479 |
PageCount | 6 |
ParticipantIDs | proquest_miscellaneous_3034248384 crossref_primary_10_1016_j_jse_2024_02_029 pubmed_primary_38580067 elsevier_sciencedirect_doi_10_1016_j_jse_2024_02_029 |
PublicationCentury | 2000 |
PublicationDate | 2024-09-01 |
PublicationDateYYYYMMDD | 2024-09-01 |
PublicationDate_xml | – month: 09 year: 2024 text: 2024-09-01 day: 01 |
PublicationDecade | 2020 |
PublicationPlace | United States |
PublicationPlace_xml | – name: United States |
PublicationTitle | Journal of shoulder and elbow surgery |
PublicationTitleAlternate | J Shoulder Elbow Surg |
PublicationYear | 2024 |
Publisher | Elsevier Inc |
Publisher_xml | – name: Elsevier Inc |
References | Hofmann, Guerra, Le, Wong, Hofmann, Mayfield (bib14) 2023; 27 Smith (bib30) 2020; 15 Giray (bib9) 2023; 51 Kung, Cheatham, Medenilla, Sillos, De Leon, Elepaño (bib21) 2023; 2 Cohen, Puntonet, Sanchez, Kierszbaum, Crema, Soyer (bib5) 2022; 33 Helm, Swiergosz, Haeberle, Karnuta, Schaffer, Krebs (bib13) 2020; 13 Stajduhar, Mamula, Miletić, Ünal (bib32) 2017; 140 (bib27) 2023 Lum (bib24) 2023; 481 Hurley, Crook, Lorentz, Danilkowicz, Lau, Taylor (bib16) 2023; S0749-8063 Kung, Marshall, Gauthier, Gonzalez, Jackson (bib20) 2023; 8 Aloimonos, Rosenfeld (bib2) 1991; 253 Tharakan, Klein, Bartlett, Atlas, Parada, Cohn (bib33) 2024; S1058-2746 Dave, Athaluri, Singh (bib7) 2023; 6 Jones, Sharma, Hotchkiss, Sperling, Hamburger, Ledig (bib17) 2020; 3 Kirchner, Kim, Weddle, Bible (bib18) 2023; 481 Yao, Chepelev, Nisha, Sathiadoss, Rybicki, Sheikh (bib34) 2022; 51 McLendon, Christmas, Simon, Plummer, Hunt, Ahmed (bib25) 2021; 6 Ashinsky, Bouhrara, Coletta, Lehallier, Urish, Lin (bib3) 2017; 35 Guerrero, Asaad, Rajesh, Hassan, Butler (bib10) 2023; 89 Lim, Wong, Yip, Chang, Chin, Teo (bib23) 2022; 39 St Mart, Goh, Liew, Shah, Sinha (bib31) 2023; 99 Daher, Koa, Boufadel, Singh, Fares, Abboud (bib6) 2023; 7 Van Riel, Auwerx, Debbaut, Van Hees, Schoenmakers (bib29) 2017; 1 Hasty, Garbalosa, Barbato, Valdes, Powers, Hernandez (bib11) 2014; 114 Gilson, Safranek, Huang, Socrates, Chi, Taylor (bib8) 2023; 9 Kothari, Moolani (bib19) 2015; 63 Kunze, Jang, Li, Pareek, Finocchiaro, Fu (bib22) 2023; 32 Hawayek, AbouElKhir (bib12) 2023; 6 Pedoia, Haefeli, Morioka, Teng, Nardo, Souza (bib28) 2018; 47 Nwachukwu, Beck, Lee, Cancienne, Waterman, Paul (bib26) 2020; 48 Ali, Tang, Connolly, Zadnik Sullivan, Shin, Fridley (bib1) 2023; 93 Humar, Asaad, Bengur, Nguyen (bib15) 2023; 4 Chung, Han, Lee, Oh, Kim, Yoon (bib4) 2018; 89 Kothari (10.1016/j.jse.2024.02.029_bib19) 2015; 63 Lim (10.1016/j.jse.2024.02.029_bib23) 2022; 39 Hasty (10.1016/j.jse.2024.02.029_bib11) 2014; 114 Stajduhar (10.1016/j.jse.2024.02.029_bib32) 2017; 140 Jones (10.1016/j.jse.2024.02.029_bib17) 2020; 3 Kung (10.1016/j.jse.2024.02.029_bib20) 2023; 8 (10.1016/j.jse.2024.02.029_bib27) 2023 Daher (10.1016/j.jse.2024.02.029_bib6) 2023; 7 Helm (10.1016/j.jse.2024.02.029_bib13) 2020; 13 Kunze (10.1016/j.jse.2024.02.029_bib22) 2023; 32 Ali (10.1016/j.jse.2024.02.029_bib1) 2023; 93 Guerrero (10.1016/j.jse.2024.02.029_bib10) 2023; 89 Ashinsky (10.1016/j.jse.2024.02.029_bib3) 2017; 35 Aloimonos (10.1016/j.jse.2024.02.029_bib2) 1991; 253 Pedoia (10.1016/j.jse.2024.02.029_bib28) 2018; 47 Nwachukwu (10.1016/j.jse.2024.02.029_bib26) 2020; 48 Tharakan (10.1016/j.jse.2024.02.029_bib33) 2024; S1058-2746 Humar (10.1016/j.jse.2024.02.029_bib15) 2023; 4 St Mart (10.1016/j.jse.2024.02.029_bib31) 2023; 99 Yao (10.1016/j.jse.2024.02.029_bib34) 2022; 51 McLendon (10.1016/j.jse.2024.02.029_bib25) 2021; 6 Dave (10.1016/j.jse.2024.02.029_bib7) 2023; 6 Lum (10.1016/j.jse.2024.02.029_bib24) 2023; 481 Hofmann (10.1016/j.jse.2024.02.029_bib14) 2023; 27 Giray (10.1016/j.jse.2024.02.029_bib9) 2023; 51 Kung (10.1016/j.jse.2024.02.029_bib21) 2023; 2 Cohen (10.1016/j.jse.2024.02.029_bib5) 2022; 33 Hawayek (10.1016/j.jse.2024.02.029_bib12) 2023; 6 Kirchner (10.1016/j.jse.2024.02.029_bib18) 2023; 481 Smith (10.1016/j.jse.2024.02.029_bib30) 2020; 15 Gilson (10.1016/j.jse.2024.02.029_bib8) 2023; 9 Hurley (10.1016/j.jse.2024.02.029_bib16) 2023; S0749-8063 Chung (10.1016/j.jse.2024.02.029_bib4) 2018; 89 Van Riel (10.1016/j.jse.2024.02.029_bib29) 2017; 1 |
References_xml | – volume: 35 start-page: 2243 year: 2017 end-page: 2250 ident: bib3 article-title: Predicting early symptomatic osteoarthritis in the human knee using machine learning classification of magnetic resonance images from the osteoarthritis initiative publication-title: J Orthop Res contributor: fullname: Lin – volume: 51 start-page: 2629 year: 2023 end-page: 2633 ident: bib9 article-title: Prompt engineering with ChatGPT: a guide for academic writers publication-title: Ann Biomed Eng contributor: fullname: Giray – volume: 51 start-page: 1765 year: 2022 end-page: 1775 ident: bib34 article-title: Evaluation of a deep learning method for the automated detection of supraspinatus tears on MRI publication-title: Skeletal Radiol contributor: fullname: Sheikh – volume: 39 start-page: 38 year: 2022 end-page: 45 ident: bib23 article-title: Online health information-seeking behaviour of patients attending a primary care clinic in Malaysia: a cross-sectional study publication-title: Fam Pract contributor: fullname: Teo – volume: 48 start-page: 415 year: 2020 end-page: 423 ident: bib26 article-title: Application of machine learning for predicting clinically meaningful outcome after arthroscopic femoroacetabular impingement surgery publication-title: Am J Sports Med contributor: fullname: Paul – volume: 3 start-page: 1 year: 2020 end-page: 6 ident: bib17 article-title: Assessment of a deep-learning system for fracture detection in musculoskeletal radiographs publication-title: NPJ Digit Med contributor: fullname: Ledig – volume: 32 start-page: 2115 year: 2023 end-page: 2122 ident: bib22 article-title: Artificial intelligence for automated identification of total shoulder arthroplasty implants publication-title: J Shoulder Elbow Surg contributor: fullname: Fu – volume: 253 start-page: 1249 year: 1991 end-page: 1254 ident: bib2 article-title: Computer vision publication-title: Science contributor: fullname: Rosenfeld – volume: 114 start-page: 368 year: 2014 end-page: 373 ident: bib11 article-title: Wikipedia vs peer-reviewed medical literature for information about the 10 most costly medical conditions publication-title: J Am Osteopath Assoc contributor: fullname: Hernandez – volume: 33 start-page: 3974 year: 2022 end-page: 3983 ident: bib5 article-title: Artificial intelligence vs. radiologist: accuracy of wrist fracture detection on radiographs publication-title: Eur Radiol contributor: fullname: Soyer – volume: 27 start-page: 1 year: 2023 end-page: 5 ident: bib14 article-title: The rapid development of artificial intelligence: GPT-4’s performance on orthopedic surgery board questions publication-title: Orthopedics contributor: fullname: Mayfield – volume: 481 start-page: 2260 year: 2023 end-page: 2267 ident: bib18 article-title: Can artificial intelligence improve the readability of patient education materials? publication-title: Clin Orthop Relat Res contributor: fullname: Bible – volume: 63 start-page: 267 year: 2015 end-page: 269 ident: bib19 article-title: Reliability of “Google” for obtaining medical information publication-title: Indian J Ophthalmol contributor: fullname: Moolani – volume: 4 year: 2023 ident: bib15 article-title: ChatGPT is equivalent to first year plastic surgery residents: evaluation of ChatGPT on the plastic surgery in-service exam publication-title: Aesthet Surg J contributor: fullname: Nguyen – volume: 99 start-page: 687 year: 2023 end-page: 694 ident: bib31 article-title: Artificial intelligence in orthopaedics surgery: transforming technological innovation in patient care and surgical training publication-title: Postgrad Med J contributor: fullname: Sinha – volume: 8 year: 2023 ident: bib20 article-title: Evaluating ChatGPT performance on the orthopaedic in-training examination publication-title: JB JS Open Access contributor: fullname: Jackson – volume: 481 start-page: 1623 year: 2023 end-page: 1630 ident: bib24 article-title: Can artificial intelligence pass the American Board of Orthopaedic Surgery examination? Orthopaedic residents versus ChatGPT publication-title: Clin Orthop Relat Res contributor: fullname: Lum – volume: 9 year: 2023 ident: bib8 article-title: How Does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment publication-title: JMIR Med Educ contributor: fullname: Taylor – volume: 2 year: 2023 ident: bib21 article-title: Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models publication-title: PLOS Digit Health contributor: fullname: Elepaño – volume: 89 start-page: 49 year: 2023 end-page: 54 ident: bib10 article-title: Advancing surgical education: the Use of artificial intelligence in surgical training publication-title: Am Surg contributor: fullname: Butler – volume: 6 year: 2021 ident: bib25 article-title: Machine learning can predict level of improvement in shoulder arthroplasty publication-title: JB JS Open Access contributor: fullname: Ahmed – volume: 47 start-page: 78 year: 2018 end-page: 90 ident: bib28 article-title: MRI and biomechanics multidimensional data analysis reveals R2-R1ρ as an early predictor of cartilage lesion progression in knee osteoarthritis publication-title: J Magn Reson Imag contributor: fullname: Souza – volume: 13 start-page: 69 year: 2020 end-page: 76 ident: bib13 article-title: Machine learning and artificial intelligence: definitions, applications, and future directions publication-title: Curr Rev Musculoskelet Med contributor: fullname: Krebs – volume: S0749-8063 start-page: 00642 year: 2023 ident: bib16 article-title: Evaluation high-quality of information from ChatGPT (artificial intelligence-large language model) artificial intelligence on shoulder stabilization surgery publication-title: Arthroscopy contributor: fullname: Taylor – volume: 89 start-page: 468 year: 2018 end-page: 473 ident: bib4 article-title: Automated detection and classification of the proximal humerus fracture by using deep learning algorithm publication-title: Acta Orthop contributor: fullname: Yoon – volume: 6 year: 2023 ident: bib7 article-title: ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations publication-title: Front Artif Intell contributor: fullname: Singh – volume: 6 year: 2023 ident: bib12 article-title: Problems with medical claims that artificial intelligence (AI) and blockchain can fix publication-title: Blockchain Healthc Today contributor: fullname: AbouElKhir – volume: 15 year: 2020 ident: bib30 article-title: Situating wikipedia as a health information resource in various contexts: a scoping review publication-title: PLoS One contributor: fullname: Smith – volume: 93 start-page: 1353 year: 2023 ident: bib1 article-title: Performance of ChatGPT and GPT-4 on neurosurgery written board examinations publication-title: Neurosurgery contributor: fullname: Fridley – volume: 7 start-page: 2534 year: 2023 end-page: 2541 ident: bib6 article-title: Breaking barriers: can ChatGPT compete with a shoulder and elbow specialist in diagnosis and management? publication-title: JSES Int contributor: fullname: Abboud – volume: S1058-2746 start-page: 00899 year: 2024 ident: bib33 article-title: Do ChatGPT and Google differ in answers to commonly Asked patient questions regarding total shoulder and total elbow arthroplasty? publication-title: J Shoulder Elbow Surg contributor: fullname: Cohn – year: 2023 ident: bib27 article-title: GPT-4 technical report – volume: 140 start-page: 151 year: 2017 end-page: 164 ident: bib32 article-title: Semi-automated detection of anterior cruciate ligament injury from MRI publication-title: Comput Methods Programs Biomed contributor: fullname: Ünal – volume: 1 year: 2017 ident: bib29 article-title: The effect of Dr Google on doctor–patient encounters in primary care: a quantitative, observational, cross-sectional study publication-title: BJGP Open contributor: fullname: Schoenmakers – volume: 89 start-page: 468 year: 2018 ident: 10.1016/j.jse.2024.02.029_bib4 article-title: Automated detection and classification of the proximal humerus fracture by using deep learning algorithm publication-title: Acta Orthop doi: 10.1080/17453674.2018.1453714 contributor: fullname: Chung – volume: 32 start-page: 2115 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib22 article-title: Artificial intelligence for automated identification of total shoulder arthroplasty implants publication-title: J Shoulder Elbow Surg doi: 10.1016/j.jse.2023.03.028 contributor: fullname: Kunze – volume: 47 start-page: 78 year: 2018 ident: 10.1016/j.jse.2024.02.029_bib28 article-title: MRI and biomechanics multidimensional data analysis reveals R2-R1ρ as an early predictor of cartilage lesion progression in knee osteoarthritis publication-title: J Magn Reson Imag doi: 10.1002/jmri.25750 contributor: fullname: Pedoia – volume: 4 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib15 article-title: ChatGPT is equivalent to first year plastic surgery residents: evaluation of ChatGPT on the plastic surgery in-service exam publication-title: Aesthet Surg J contributor: fullname: Humar – volume: 481 start-page: 2260 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib18 article-title: Can artificial intelligence improve the readability of patient education materials? publication-title: Clin Orthop Relat Res doi: 10.1097/CORR.0000000000002668 contributor: fullname: Kirchner – volume: 1 year: 2017 ident: 10.1016/j.jse.2024.02.029_bib29 article-title: The effect of Dr Google on doctor–patient encounters in primary care: a quantitative, observational, cross-sectional study publication-title: BJGP Open doi: 10.3399/bjgpopen17X100833 contributor: fullname: Van Riel – volume: S1058-2746 start-page: 00899 year: 2024 ident: 10.1016/j.jse.2024.02.029_bib33 article-title: Do ChatGPT and Google differ in answers to commonly Asked patient questions regarding total shoulder and total elbow arthroplasty? publication-title: J Shoulder Elbow Surg contributor: fullname: Tharakan – volume: S0749-8063 start-page: 00642 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib16 article-title: Evaluation high-quality of information from ChatGPT (artificial intelligence-large language model) artificial intelligence on shoulder stabilization surgery publication-title: Arthroscopy contributor: fullname: Hurley – volume: 8 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib20 article-title: Evaluating ChatGPT performance on the orthopaedic in-training examination publication-title: JB JS Open Access contributor: fullname: Kung – volume: 33 start-page: 3974 year: 2022 ident: 10.1016/j.jse.2024.02.029_bib5 article-title: Artificial intelligence vs. radiologist: accuracy of wrist fracture detection on radiographs publication-title: Eur Radiol doi: 10.1007/s00330-022-09349-3 contributor: fullname: Cohen – volume: 63 start-page: 267 year: 2015 ident: 10.1016/j.jse.2024.02.029_bib19 article-title: Reliability of “Google” for obtaining medical information publication-title: Indian J Ophthalmol doi: 10.4103/0301-4738.156934 contributor: fullname: Kothari – volume: 3 start-page: 1 year: 2020 ident: 10.1016/j.jse.2024.02.029_bib17 article-title: Assessment of a deep-learning system for fracture detection in musculoskeletal radiographs publication-title: NPJ Digit Med doi: 10.1038/s41746-020-00352-w contributor: fullname: Jones – volume: 6 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib7 article-title: ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations publication-title: Front Artif Intell doi: 10.3389/frai.2023.1169595 contributor: fullname: Dave – volume: 51 start-page: 1765 year: 2022 ident: 10.1016/j.jse.2024.02.029_bib34 article-title: Evaluation of a deep learning method for the automated detection of supraspinatus tears on MRI publication-title: Skeletal Radiol doi: 10.1007/s00256-022-04008-6 contributor: fullname: Yao – volume: 481 start-page: 1623 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib24 article-title: Can artificial intelligence pass the American Board of Orthopaedic Surgery examination? Orthopaedic residents versus ChatGPT publication-title: Clin Orthop Relat Res doi: 10.1097/CORR.0000000000002704 contributor: fullname: Lum – volume: 27 start-page: 1 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib14 article-title: The rapid development of artificial intelligence: GPT-4’s performance on orthopedic surgery board questions publication-title: Orthopedics contributor: fullname: Hofmann – year: 2023 ident: 10.1016/j.jse.2024.02.029_bib27 – volume: 9 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib8 article-title: How Does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment publication-title: JMIR Med Educ doi: 10.2196/45312 contributor: fullname: Gilson – volume: 2 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib21 article-title: Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models publication-title: PLOS Digit Health doi: 10.1371/journal.pdig.0000198 contributor: fullname: Kung – volume: 7 start-page: 2534 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib6 article-title: Breaking barriers: can ChatGPT compete with a shoulder and elbow specialist in diagnosis and management? publication-title: JSES Int doi: 10.1016/j.jseint.2023.07.018 contributor: fullname: Daher – volume: 35 start-page: 2243 year: 2017 ident: 10.1016/j.jse.2024.02.029_bib3 article-title: Predicting early symptomatic osteoarthritis in the human knee using machine learning classification of magnetic resonance images from the osteoarthritis initiative publication-title: J Orthop Res doi: 10.1002/jor.23519 contributor: fullname: Ashinsky – volume: 6 year: 2021 ident: 10.1016/j.jse.2024.02.029_bib25 article-title: Machine learning can predict level of improvement in shoulder arthroplasty publication-title: JB JS Open Access contributor: fullname: McLendon – volume: 6 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib12 article-title: Problems with medical claims that artificial intelligence (AI) and blockchain can fix publication-title: Blockchain Healthc Today doi: 10.30953/bhty.v6.273 contributor: fullname: Hawayek – volume: 253 start-page: 1249 year: 1991 ident: 10.1016/j.jse.2024.02.029_bib2 article-title: Computer vision publication-title: Science doi: 10.1126/science.1891713 contributor: fullname: Aloimonos – volume: 39 start-page: 38 year: 2022 ident: 10.1016/j.jse.2024.02.029_bib23 article-title: Online health information-seeking behaviour of patients attending a primary care clinic in Malaysia: a cross-sectional study publication-title: Fam Pract doi: 10.1093/fampra/cmab099 contributor: fullname: Lim – volume: 48 start-page: 415 year: 2020 ident: 10.1016/j.jse.2024.02.029_bib26 article-title: Application of machine learning for predicting clinically meaningful outcome after arthroscopic femoroacetabular impingement surgery publication-title: Am J Sports Med doi: 10.1177/0363546519892905 contributor: fullname: Nwachukwu – volume: 51 start-page: 2629 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib9 article-title: Prompt engineering with ChatGPT: a guide for academic writers publication-title: Ann Biomed Eng doi: 10.1007/s10439-023-03272-4 contributor: fullname: Giray – volume: 13 start-page: 69 year: 2020 ident: 10.1016/j.jse.2024.02.029_bib13 article-title: Machine learning and artificial intelligence: definitions, applications, and future directions publication-title: Curr Rev Musculoskelet Med doi: 10.1007/s12178-020-09600-8 contributor: fullname: Helm – volume: 15 year: 2020 ident: 10.1016/j.jse.2024.02.029_bib30 article-title: Situating wikipedia as a health information resource in various contexts: a scoping review publication-title: PLoS One contributor: fullname: Smith – volume: 89 start-page: 49 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib10 article-title: Advancing surgical education: the Use of artificial intelligence in surgical training publication-title: Am Surg doi: 10.1177/00031348221101503 contributor: fullname: Guerrero – volume: 93 start-page: 1353 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib1 article-title: Performance of ChatGPT and GPT-4 on neurosurgery written board examinations publication-title: Neurosurgery doi: 10.1227/neu.0000000000002632 contributor: fullname: Ali – volume: 114 start-page: 368 year: 2014 ident: 10.1016/j.jse.2024.02.029_bib11 article-title: Wikipedia vs peer-reviewed medical literature for information about the 10 most costly medical conditions publication-title: J Am Osteopath Assoc contributor: fullname: Hasty – volume: 99 start-page: 687 year: 2023 ident: 10.1016/j.jse.2024.02.029_bib31 article-title: Artificial intelligence in orthopaedics surgery: transforming technological innovation in patient care and surgical training publication-title: Postgrad Med J doi: 10.1136/postgradmedj-2022-141596 contributor: fullname: St Mart – volume: 140 start-page: 151 year: 2017 ident: 10.1016/j.jse.2024.02.029_bib32 article-title: Semi-automated detection of anterior cruciate ligament injury from MRI publication-title: Comput Methods Programs Biomed doi: 10.1016/j.cmpb.2016.12.006 contributor: fullname: Stajduhar |
SSID | ssj0009402 |
Score | 2.5055637 |
Snippet | While multiple studies have tested the ability of large language models (LLMs), such as ChatGPT, to pass standardized medical exams at different levels of... While multiple studies have tested the ability of large language models (LLM), such as ChatGPT, to pass standardized medical exams at different levels of... |
SourceID | proquest crossref pubmed elsevier |
SourceType | Aggregation Database Index Database Publisher |
StartPage | 1888 |
SubjectTerms | American Shoulder and Elbow Surgeons (ASES) artificial intelligence (AI) ChatGPT elbow machine learning maintenance of certification (MOC) shoulder |
Title | ChatGPT performance on the American Shoulder and Elbow Surgeons maintenance of certification exam |
URI | https://dx.doi.org/10.1016/j.jse.2024.02.029 https://www.ncbi.nlm.nih.gov/pubmed/38580067 https://www.proquest.com/docview/3034248384 |
Volume | 33 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1LSwMxEA6lXryI4qs-SgRPwtptHrvJsZTWqliEttBbSDYJbbHb0gd68reb7EMR1IOwl10SNswMM9_ufDMDwDVXiDinZwOtaBIQpsJAOdMIbKRoaKRF1PpC4ad-1BuRhzEdV0C7rIXxtMrC9-c-PfPWxZNGIc3GcjptDBwyYO6bKvIsSIdSfMdPQkM_xeD2_YvmwUnBO6TOItzqMrOZcbxma98pE5GsbWeGMn-MTb9hzywGdffBXgEeYSs_3wGomPQQyPZEbu6eh3D5VQMAFyl0yA6W-Rg4mPhR1mYFZaph50UtXuHAF0Q7m4Nz6XtGpPk-CxPPtLbFrzxo3uT8CIy6nWG7FxSDE4IEYboJqFYWKRlJSk2MsY4lNghJbjnV1FiWkKakmGKLmpZyLkNuQmW5tgzxxAV1fAyq6SI1pwCyUGqLDI8NUYRpt5JH2qpYMRslWsc1cFOKTCzz_hiiJI7NhJOv8PIVIXIXrwFSClV8U7Jw_vuvbVelAoQzfp_RkKlZbNcC-waGhGFGauAk18znKXzG08fis_-99Bzs-rucTnYBqpvV1lw6_LFR9czA6mCndf_Y638AstzavA |
link.rule.ids | 314,780,784,4502,24116,27924,27925,45585,45679 |
linkProvider | Elsevier |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1LSwMxEA4-DnoRxVd9RvAkrF3z2E2OUqz1idAWvIVkk9AW3Rat6Mnf7mQfiqAehD3tJmyYfMx8u_PNBKFDaQgDp-cja3gWMWHiyAA0Ip8YHjvtCfehUPjmNun02eU9v59BrboWJsgqK99f-vTCW1d3mpU1m5PhsNkFZiDgmyoJKkhgKXQWzTMO7BdAffz-pfOQrBIecoAEDK9Tm4XIa_QcWmUSVvTtLGjmj8HpN_JZBKH2Mlqq2CM-LRe4gmZcvop0a6Cn53c9PPkqAsDjHAO1w3VCBncH4Sxr94R1bvHZgxm_4m6oiAbQ4Ucdmkbk5TyPsyC19tW_POze9OMa6rfPeq1OVJ2cEGWE8mnErfHE6ERz7lJKbaqpI0RLL7nlzouMnWhOOfXkxHMpdSxdbLy0XhCZQVSn62guH-duE2ERa-uJk6ljhgkLI2VivUmN8ElmbdpAR7XJ1KRskKFq5dhIgX1VsK-KCVyygVhtVPVtlxU48L-mHdQboAD9IaWhczd-eVY0dDBkggrWQBvlznyuIqQ8QzDe-t9L99FCp3dzra4vbq-20WJ4UmrLdtDc9OnF7QIZmZq9Amwf9yDcWQ |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ChatGPT+performance+on+the+American+Shoulder+and+Elbow+Surgeons+maintenance+of+certification+exam&rft.jtitle=Journal+of+shoulder+and+elbow+surgery&rft.au=Fiedler%2C+Benjamin&rft.au=Azua%2C+Eric+N&rft.au=Phillips%2C+Todd&rft.au=Ahmed%2C+Adil+Shahzad&rft.date=2024-09-01&rft.issn=1532-6500&rft.eissn=1532-6500&rft.volume=33&rft.issue=9&rft.spage=1888&rft_id=info:doi/10.1016%2Fj.jse.2024.02.029&rft.externalDBID=NO_FULL_TEXT |
thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1058-2746&client=summon |
thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1058-2746&client=summon |
thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1058-2746&client=summon |