ChatGPT in Occupational Medicine: A Comparative Study with Human Experts

• Published in: Bioengineering (Basel) Vol. 11; no. 1; p. 57
• Main Authors: Padovan, Martina, Cosci, Bianca, Petillo, Armando, Nerli, Gianluca, Porciatti, Francesco, Scarinci, Sergio, Carlucci, Francesco, Dell’Amico, Letizia, Meliani, Niccolò, Necciari, Gabriele, Lucisano, Vincenzo Carmelo, Marino, Riccardo, Foddis, Rudy, Palla, Alessandro
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 01.01.2024
• Subjects: Accuracy; Application programming interface; Artificial intelligence; Chatbots; ChatGPT; Comparative studies; Decision making; digital health; health promotion; large language model; Legislation; Medical personnel; Medical screening; Medicine; Medicine, Industrial; Natural language; Occupational health; occupational health and safety; Physicians; Questions; R&D; Research & development; Semantics; Teams; Technology application; Usability; United States; Italy; health promotion; digital health; large language model; occupational health and safety; ChatGPT; artificial intelligence
• ISSN: 2306-5354; 2306-5354
• DOI: 10.3390/bioengineering11010057

The objective of this study is to evaluate ChatGPT’s accuracy and reliability in answering complex medical questions related to occupational health and explore the implications and limitations of AI in occupational health medicine. The study also provides recommendations for future research in this area and informs decision-makers about AI’s impact on healthcare. A group of physicians was enlisted to create a dataset of questions and answers on Italian occupational medicine legislation. The physicians were divided into two teams, and each team member was assigned a different subject area. ChatGPT was used to generate answers for each question, with/without legislative context. The two teams then evaluated human and AI-generated answers blind, with each group reviewing the other group’s work. Occupational physicians outperformed ChatGPT in generating accurate questions on a 5-point Likert score, while the answers provided by ChatGPT with access to legislative texts were comparable to those of professional doctors. Still, we found that users tend to prefer answers generated by humans, indicating that while ChatGPT is useful, users still value the opinions of occupational medicine professionals.