Identification of Online Health Information Using Large Pretrained Language Models: Mixed Methods Study
Online health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or unverified claims. Such misinformation can significantly influence public health decisions and pose serious challenges to health care systems. With advanc...
Saved in:
| Published in | Journal of medical Internet research Vol. 27; no. 5; p. e70733 |
|---|---|
| Main Authors | , , , , , |
| Format | Journal Article |
| Language | English |
| Published |
Canada
Journal of Medical Internet Research
14.05.2025
Gunther Eysenbach MD MPH, Associate Professor JMIR Publications |
| Subjects | |
| Online Access | Get full text |
Cover
Loading…
| Abstract | Online health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or unverified claims. Such misinformation can significantly influence public health decisions and pose serious challenges to health care systems. With advances in artificial intelligence and natural language processing, pretrained large language models (LLMs) have shown promise in identifying and distinguishing misleading health information, although their effectiveness in this area remains underexplored.
This study aimed to evaluate the performance of 4 mainstream LLMs (ChatGPT-3.5, ChatGPT-4, Ernie Bot, and iFLYTEK Spark) in the identification of online health information, providing empirical evidence for their practical application in this field.
Web scraping was used to collect data from rumor-refuting websites, resulting in 2708 samples of online health information, including both true and false claims. The 4 LLMs' application programming interfaces were used for authenticity verification, with expert results as benchmarks. Model performance was evaluated using semantic similarity, accuracy, recall, F
-score, content analysis, and credibility.
This study found that the 4 models performed well in identifying online health information. Among them, ChatGPT-4 achieved the highest accuracy at 87.27%, followed by Ernie Bot at 87.25%, iFLYTEK Spark at 87%, and ChatGPT-3.5 at 81.82%. Furthermore, text length and semantic similarity analysis showed that Ernie Bot had the highest similarity to expert texts, whereas ChatGPT-4 showed good overall consistency in its explanations. In addition, the credibility assessment results indicated that ChatGPT-4 provided the most reliable evaluations. Further analysis suggested that the highest misjudgment probabilities with respect to the LLMs occurred within the topics of food and maternal-infant nutrition management and nutritional science and food controversies. Overall, the research suggests that LLMs have potential in online health information identification; however, their understanding of certain specialized health topics may require further improvement.
The results demonstrate that, while these models show potential in providing assistance, their performance varies significantly in terms of accuracy, semantic understanding, and cultural adaptability. The principal findings highlight the models' ability to generate accessible and context-aware explanations; however, they fall short in areas requiring specialized medical knowledge or updated data, particularly for emerging health issues and context-sensitive scenarios. Significant discrepancies were observed in the models' ability to distinguish scientifically verified knowledge from popular misconceptions and in their stability when processing complex linguistic and cultural contexts. These challenges reveal the importance of refining training methodologies to improve the models' reliability and adaptability. Future research should focus on enhancing the models' capability to manage nuanced health topics and diverse cultural and linguistic nuances, thereby facilitating their broader adoption as reliable tools for online health information identification. |
|---|---|
| AbstractList | Online health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or unverified claims. Such misinformation can significantly influence public health decisions and pose serious challenges to health care systems. With advances in artificial intelligence and natural language processing, pretrained large language models (LLMs) have shown promise in identifying and distinguishing misleading health information, although their effectiveness in this area remains underexplored.
This study aimed to evaluate the performance of 4 mainstream LLMs (ChatGPT-3.5, ChatGPT-4, Ernie Bot, and iFLYTEK Spark) in the identification of online health information, providing empirical evidence for their practical application in this field.
Web scraping was used to collect data from rumor-refuting websites, resulting in 2708 samples of online health information, including both true and false claims. The 4 LLMs' application programming interfaces were used for authenticity verification, with expert results as benchmarks. Model performance was evaluated using semantic similarity, accuracy, recall, F
-score, content analysis, and credibility.
This study found that the 4 models performed well in identifying online health information. Among them, ChatGPT-4 achieved the highest accuracy at 87.27%, followed by Ernie Bot at 87.25%, iFLYTEK Spark at 87%, and ChatGPT-3.5 at 81.82%. Furthermore, text length and semantic similarity analysis showed that Ernie Bot had the highest similarity to expert texts, whereas ChatGPT-4 showed good overall consistency in its explanations. In addition, the credibility assessment results indicated that ChatGPT-4 provided the most reliable evaluations. Further analysis suggested that the highest misjudgment probabilities with respect to the LLMs occurred within the topics of food and maternal-infant nutrition management and nutritional science and food controversies. Overall, the research suggests that LLMs have potential in online health information identification; however, their understanding of certain specialized health topics may require further improvement.
The results demonstrate that, while these models show potential in providing assistance, their performance varies significantly in terms of accuracy, semantic understanding, and cultural adaptability. The principal findings highlight the models' ability to generate accessible and context-aware explanations; however, they fall short in areas requiring specialized medical knowledge or updated data, particularly for emerging health issues and context-sensitive scenarios. Significant discrepancies were observed in the models' ability to distinguish scientifically verified knowledge from popular misconceptions and in their stability when processing complex linguistic and cultural contexts. These challenges reveal the importance of refining training methodologies to improve the models' reliability and adaptability. Future research should focus on enhancing the models' capability to manage nuanced health topics and diverse cultural and linguistic nuances, thereby facilitating their broader adoption as reliable tools for online health information identification. Online health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or unverified claims. Such misinformation can significantly influence public health decisions and pose serious challenges to health care systems. With advances in artificial intelligence and natural language processing, pretrained large language models (LLMs) have shown promise in identifying and distinguishing misleading health information, although their effectiveness in this area remains underexplored.BACKGROUNDOnline health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or unverified claims. Such misinformation can significantly influence public health decisions and pose serious challenges to health care systems. With advances in artificial intelligence and natural language processing, pretrained large language models (LLMs) have shown promise in identifying and distinguishing misleading health information, although their effectiveness in this area remains underexplored.This study aimed to evaluate the performance of 4 mainstream LLMs (ChatGPT-3.5, ChatGPT-4, Ernie Bot, and iFLYTEK Spark) in the identification of online health information, providing empirical evidence for their practical application in this field.OBJECTIVEThis study aimed to evaluate the performance of 4 mainstream LLMs (ChatGPT-3.5, ChatGPT-4, Ernie Bot, and iFLYTEK Spark) in the identification of online health information, providing empirical evidence for their practical application in this field.Web scraping was used to collect data from rumor-refuting websites, resulting in 2708 samples of online health information, including both true and false claims. The 4 LLMs' application programming interfaces were used for authenticity verification, with expert results as benchmarks. Model performance was evaluated using semantic similarity, accuracy, recall, F1-score, content analysis, and credibility.METHODSWeb scraping was used to collect data from rumor-refuting websites, resulting in 2708 samples of online health information, including both true and false claims. The 4 LLMs' application programming interfaces were used for authenticity verification, with expert results as benchmarks. Model performance was evaluated using semantic similarity, accuracy, recall, F1-score, content analysis, and credibility.This study found that the 4 models performed well in identifying online health information. Among them, ChatGPT-4 achieved the highest accuracy at 87.27%, followed by Ernie Bot at 87.25%, iFLYTEK Spark at 87%, and ChatGPT-3.5 at 81.82%. Furthermore, text length and semantic similarity analysis showed that Ernie Bot had the highest similarity to expert texts, whereas ChatGPT-4 showed good overall consistency in its explanations. In addition, the credibility assessment results indicated that ChatGPT-4 provided the most reliable evaluations. Further analysis suggested that the highest misjudgment probabilities with respect to the LLMs occurred within the topics of food and maternal-infant nutrition management and nutritional science and food controversies. Overall, the research suggests that LLMs have potential in online health information identification; however, their understanding of certain specialized health topics may require further improvement.RESULTSThis study found that the 4 models performed well in identifying online health information. Among them, ChatGPT-4 achieved the highest accuracy at 87.27%, followed by Ernie Bot at 87.25%, iFLYTEK Spark at 87%, and ChatGPT-3.5 at 81.82%. Furthermore, text length and semantic similarity analysis showed that Ernie Bot had the highest similarity to expert texts, whereas ChatGPT-4 showed good overall consistency in its explanations. In addition, the credibility assessment results indicated that ChatGPT-4 provided the most reliable evaluations. Further analysis suggested that the highest misjudgment probabilities with respect to the LLMs occurred within the topics of food and maternal-infant nutrition management and nutritional science and food controversies. Overall, the research suggests that LLMs have potential in online health information identification; however, their understanding of certain specialized health topics may require further improvement.The results demonstrate that, while these models show potential in providing assistance, their performance varies significantly in terms of accuracy, semantic understanding, and cultural adaptability. The principal findings highlight the models' ability to generate accessible and context-aware explanations; however, they fall short in areas requiring specialized medical knowledge or updated data, particularly for emerging health issues and context-sensitive scenarios. Significant discrepancies were observed in the models' ability to distinguish scientifically verified knowledge from popular misconceptions and in their stability when processing complex linguistic and cultural contexts. These challenges reveal the importance of refining training methodologies to improve the models' reliability and adaptability. Future research should focus on enhancing the models' capability to manage nuanced health topics and diverse cultural and linguistic nuances, thereby facilitating their broader adoption as reliable tools for online health information identification.CONCLUSIONSThe results demonstrate that, while these models show potential in providing assistance, their performance varies significantly in terms of accuracy, semantic understanding, and cultural adaptability. The principal findings highlight the models' ability to generate accessible and context-aware explanations; however, they fall short in areas requiring specialized medical knowledge or updated data, particularly for emerging health issues and context-sensitive scenarios. Significant discrepancies were observed in the models' ability to distinguish scientifically verified knowledge from popular misconceptions and in their stability when processing complex linguistic and cultural contexts. These challenges reveal the importance of refining training methodologies to improve the models' reliability and adaptability. Future research should focus on enhancing the models' capability to manage nuanced health topics and diverse cultural and linguistic nuances, thereby facilitating their broader adoption as reliable tools for online health information identification. Online health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or unverified claims. Such misinformation can significantly influence public health decisions and pose serious challenges to health care systems. With advances in artificial intelligence and natural language processing, pretrained large language models (LLMs) have shown promise in identifying and distinguishing misleading health information, although their effectiveness in this area remains underexplored. This study aimed to evaluate the performance of 4 mainstream LLMs (ChatGPT-3.5, ChatGPT-4, Ernie Bot, and iFLYTEK Spark) in the identification of online health information, providing empirical evidence for their practical application in this field. Web scraping was used to collect data from rumor-refuting websites, resulting in 2708 samples of online health information, including both true and false claims. The 4 LLMs’ application programming interfaces were used for authenticity verification, with expert results as benchmarks. Model performance was evaluated using semantic similarity, accuracy, recall, F [sub.1] -score, content analysis, and credibility. This study found that the 4 models performed well in identifying online health information. Among them, ChatGPT-4 achieved the highest accuracy at 87.27%, followed by Ernie Bot at 87.25%, iFLYTEK Spark at 87%, and ChatGPT-3.5 at 81.82%. Furthermore, text length and semantic similarity analysis showed that Ernie Bot had the highest similarity to expert texts, whereas ChatGPT-4 showed good overall consistency in its explanations. In addition, the credibility assessment results indicated that ChatGPT-4 provided the most reliable evaluations. Further analysis suggested that the highest misjudgment probabilities with respect to the LLMs occurred within the topics of food and maternal-infant nutrition management and nutritional science and food controversies . Overall, the research suggests that LLMs have potential in online health information identification; however, their understanding of certain specialized health topics may require further improvement. The results demonstrate that, while these models show potential in providing assistance, their performance varies significantly in terms of accuracy, semantic understanding, and cultural adaptability. The principal findings highlight the models’ ability to generate accessible and context-aware explanations; however, they fall short in areas requiring specialized medical knowledge or updated data, particularly for emerging health issues and context-sensitive scenarios. Significant discrepancies were observed in the models’ ability to distinguish scientifically verified knowledge from popular misconceptions and in their stability when processing complex linguistic and cultural contexts. These challenges reveal the importance of refining training methodologies to improve the models’ reliability and adaptability. Future research should focus on enhancing the models’ capability to manage nuanced health topics and diverse cultural and linguistic nuances, thereby facilitating their broader adoption as reliable tools for online health information identification. Background:Online health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or unverified claims. Such misinformation can significantly influence public health decisions and pose serious challenges to health care systems. With advances in artificial intelligence and natural language processing, pretrained large language models (LLMs) have shown promise in identifying and distinguishing misleading health information, although their effectiveness in this area remains underexplored.Objective:This study aimed to evaluate the performance of 4 mainstream LLMs (ChatGPT-3.5, ChatGPT-4, Ernie Bot, and iFLYTEK Spark) in the identification of online health information, providing empirical evidence for their practical application in this field.Methods:Web scraping was used to collect data from rumor-refuting websites, resulting in 2708 samples of online health information, including both true and false claims. The 4 LLMs’ application programming interfaces were used for authenticity verification, with expert results as benchmarks. Model performance was evaluated using semantic similarity, accuracy, recall, F1-score, content analysis, and credibility.Results:This study found that the 4 models performed well in identifying online health information. Among them, ChatGPT-4 achieved the highest accuracy at 87.27%, followed by Ernie Bot at 87.25%, iFLYTEK Spark at 87%, and ChatGPT-3.5 at 81.82%. Furthermore, text length and semantic similarity analysis showed that Ernie Bot had the highest similarity to expert texts, whereas ChatGPT-4 showed good overall consistency in its explanations. In addition, the credibility assessment results indicated that ChatGPT-4 provided the most reliable evaluations. Further analysis suggested that the highest misjudgment probabilities with respect to the LLMs occurred within the topics of food and maternal-infant nutrition management and nutritional science and food controversies. Overall, the research suggests that LLMs have potential in online health information identification; however, their understanding of certain specialized health topics may require further improvement.Conclusions:The results demonstrate that, while these models show potential in providing assistance, their performance varies significantly in terms of accuracy, semantic understanding, and cultural adaptability. The principal findings highlight the models’ ability to generate accessible and context-aware explanations; however, they fall short in areas requiring specialized medical knowledge or updated data, particularly for emerging health issues and context-sensitive scenarios. Significant discrepancies were observed in the models’ ability to distinguish scientifically verified knowledge from popular misconceptions and in their stability when processing complex linguistic and cultural contexts. These challenges reveal the importance of refining training methodologies to improve the models’ reliability and adaptability. Future research should focus on enhancing the models’ capability to manage nuanced health topics and diverse cultural and linguistic nuances, thereby facilitating their broader adoption as reliable tools for online health information identification. Background Online health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or unverified claims. Such misinformation can significantly influence public health decisions and pose serious challenges to health care systems. With advances in artificial intelligence and natural language processing, pretrained large language models (LLMs) have shown promise in identifying and distinguishing misleading health information, although their effectiveness in this area remains underexplored. Objective This study aimed to evaluate the performance of 4 mainstream LLMs (ChatGPT-3.5, ChatGPT-4, Ernie Bot, and iFLYTEK Spark) in the identification of online health information, providing empirical evidence for their practical application in this field. Methods Web scraping was used to collect data from rumor-refuting websites, resulting in 2708 samples of online health information, including both true and false claims. The 4 LLMs’ application programming interfaces were used for authenticity verification, with expert results as benchmarks. Model performance was evaluated using semantic similarity, accuracy, recall, F [sub.1] -score, content analysis, and credibility. Results This study found that the 4 models performed well in identifying online health information. Among them, ChatGPT-4 achieved the highest accuracy at 87.27%, followed by Ernie Bot at 87.25%, iFLYTEK Spark at 87%, and ChatGPT-3.5 at 81.82%. Furthermore, text length and semantic similarity analysis showed that Ernie Bot had the highest similarity to expert texts, whereas ChatGPT-4 showed good overall consistency in its explanations. In addition, the credibility assessment results indicated that ChatGPT-4 provided the most reliable evaluations. Further analysis suggested that the highest misjudgment probabilities with respect to the LLMs occurred within the topics of food and maternal-infant nutrition management and nutritional science and food controversies . Overall, the research suggests that LLMs have potential in online health information identification; however, their understanding of certain specialized health topics may require further improvement. Conclusions The results demonstrate that, while these models show potential in providing assistance, their performance varies significantly in terms of accuracy, semantic understanding, and cultural adaptability. The principal findings highlight the models’ ability to generate accessible and context-aware explanations; however, they fall short in areas requiring specialized medical knowledge or updated data, particularly for emerging health issues and context-sensitive scenarios. Significant discrepancies were observed in the models’ ability to distinguish scientifically verified knowledge from popular misconceptions and in their stability when processing complex linguistic and cultural contexts. These challenges reveal the importance of refining training methodologies to improve the models’ reliability and adaptability. Future research should focus on enhancing the models’ capability to manage nuanced health topics and diverse cultural and linguistic nuances, thereby facilitating their broader adoption as reliable tools for online health information identification. BackgroundOnline health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or unverified claims. Such misinformation can significantly influence public health decisions and pose serious challenges to health care systems. With advances in artificial intelligence and natural language processing, pretrained large language models (LLMs) have shown promise in identifying and distinguishing misleading health information, although their effectiveness in this area remains underexplored. ObjectiveThis study aimed to evaluate the performance of 4 mainstream LLMs (ChatGPT-3.5, ChatGPT-4, Ernie Bot, and iFLYTEK Spark) in the identification of online health information, providing empirical evidence for their practical application in this field. MethodsWeb scraping was used to collect data from rumor-refuting websites, resulting in 2708 samples of online health information, including both true and false claims. The 4 LLMs’ application programming interfaces were used for authenticity verification, with expert results as benchmarks. Model performance was evaluated using semantic similarity, accuracy, recall, F1-score, content analysis, and credibility. ResultsThis study found that the 4 models performed well in identifying online health information. Among them, ChatGPT-4 achieved the highest accuracy at 87.27%, followed by Ernie Bot at 87.25%, iFLYTEK Spark at 87%, and ChatGPT-3.5 at 81.82%. Furthermore, text length and semantic similarity analysis showed that Ernie Bot had the highest similarity to expert texts, whereas ChatGPT-4 showed good overall consistency in its explanations. In addition, the credibility assessment results indicated that ChatGPT-4 provided the most reliable evaluations. Further analysis suggested that the highest misjudgment probabilities with respect to the LLMs occurred within the topics of food and maternal-infant nutrition management and nutritional science and food controversies. Overall, the research suggests that LLMs have potential in online health information identification; however, their understanding of certain specialized health topics may require further improvement. ConclusionsThe results demonstrate that, while these models show potential in providing assistance, their performance varies significantly in terms of accuracy, semantic understanding, and cultural adaptability. The principal findings highlight the models’ ability to generate accessible and context-aware explanations; however, they fall short in areas requiring specialized medical knowledge or updated data, particularly for emerging health issues and context-sensitive scenarios. Significant discrepancies were observed in the models’ ability to distinguish scientifically verified knowledge from popular misconceptions and in their stability when processing complex linguistic and cultural contexts. These challenges reveal the importance of refining training methodologies to improve the models’ reliability and adaptability. Future research should focus on enhancing the models’ capability to manage nuanced health topics and diverse cultural and linguistic nuances, thereby facilitating their broader adoption as reliable tools for online health information identification. |
| Audience | Academic |
| Author | Tan, Dongmei Wu, Xiaoqian Liu, Ming Huang, Yi Li, Ziyu Huang, Cheng |
| AuthorAffiliation | 3 Department of Quality Management Army Medical Center Army Medical University (The Third Military Medical University) Chongqing China 1 College of Medical Informatics Chongqing Medical University Chongqing China 2 Human Resources Department Army Medical Center Army Medical University (The Third Military Medical University) Chongqing China |
| AuthorAffiliation_xml | – name: 2 Human Resources Department Army Medical Center Army Medical University (The Third Military Medical University) Chongqing China – name: 1 College of Medical Informatics Chongqing Medical University Chongqing China – name: 3 Department of Quality Management Army Medical Center Army Medical University (The Third Military Medical University) Chongqing China |
| Author_xml | – sequence: 1 givenname: Dongmei orcidid: 0009-0002-5967-8396 surname: Tan fullname: Tan, Dongmei – sequence: 2 givenname: Yi orcidid: 0009-0001-6258-801X surname: Huang fullname: Huang, Yi – sequence: 3 givenname: Ming orcidid: 0000-0002-3937-0557 surname: Liu fullname: Liu, Ming – sequence: 4 givenname: Ziyu orcidid: 0000-0003-0334-9075 surname: Li fullname: Li, Ziyu – sequence: 5 givenname: Xiaoqian orcidid: 0000-0001-6416-2445 surname: Wu fullname: Wu, Xiaoqian – sequence: 6 givenname: Cheng orcidid: 0000-0001-7937-7166 surname: Huang fullname: Huang, Cheng |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/40367512$$D View this record in MEDLINE/PubMed |
| BookMark | eNptkm9rFDEQxhep2D_2K8iCCL65mmSy2Y1vpBTbHtxRQfs6ZJPsXo69pCa7xX5753q19kTyIuGZX57MTOa4OAgxuKI4peSMUSk-1aQGeFUcUQ7NrGlqevDifFgc57wmhBEu6ZvikBMQdUXZUdHPrQuj77zRo4-hjF15EwYfXHnt9DCuynnoYtrsgrfZh75c6NS78ltyY9IIWhRCP2nUltG6IX8ul_4Xyks3rqLN5fdxsg9vi9edHrI7fdpPitvLrz8urmeLm6v5xfliZrgg48xZAwKkJKaruoqamghTQ1sBw5Shpl3DCWfCNaSWbcd5A87oyjaaS9FYS-CkmO98bdRrdZf8RqcHFbVXj0JMvdJp9GZwSjMASo12ICtOhdC1caziLbQgWiMq9Pqy87qb2g1mho1Ketgz3Y8Ev1J9vFeUUYYdBnT4-OSQ4s_J5VFtfDZuGHRwccoK8EOASUIoou__QddxSgF7hRRjICSn9V-q11iBx7_Bh83WVJ03275xkByps_9QuKzbeINz03nU9y68e1npc4l_5gSBDzvApJhzct0zQonazp96nD_4DX0vyF4 |
| Cites_doi | 10.1186/s12879-021-06670-y 10.2196/19273 10.1001/jama.2018.16865 10.2196/52399 10.1007/s10439-023-03172-7 10.1007/s11431-020-1647-3 10.1136/amiajnl-2012-001409 10.1016/j.ebiom.2023.104770 10.1197/jamia.m1687 10.1093/heapro/dag409 10.1001/jama.2024.21700 10.2753/mis0742-1222290408 10.3390/info13020083 10.1136/bmj.39489.470347.ad 10.2196/17187 10.1145/3611651 10.1002/wics.101 10.1136/bmj.328.7454.1490 10.1016/j.chb.2014.10.050 10.1038/s41598-022-11488-y 10.1001/jama.2024.6837 10.1016/j.patter.2024.100943 10.55529/jhtd.34.43.55 10.1109/jas.2023.123618 10.1016/j.socscimed.2019.112552 10.2196/19458 10.1080/28355245.2023.2263355 10.1016/j.tbench.2023.100105 10.1126/science.aao2998 10.3390/informatics11020013 10.3390/info10040150 10.1590/1806-9282.20230848 10.1093/bmb/ldab016 10.1126/science.aap9559 10.1016/j.socscimed.2007.01.012 10.2196/49771 10.2196/jmir.4018 10.1080/02604027.2019.1703158 |
| ContentType | Journal Article |
| Copyright | Dongmei Tan, Yi Huang, Ming Liu, Ziyu Li, Xiaoqian Wu, Cheng Huang. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 14.05.2025. COPYRIGHT 2025 Journal of Medical Internet Research 2025. This work is licensed under https://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. Dongmei Tan, Yi Huang, Ming Liu, Ziyu Li, Xiaoqian Wu, Cheng Huang. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 14.05.2025. 2025 |
| Copyright_xml | – notice: Dongmei Tan, Yi Huang, Ming Liu, Ziyu Li, Xiaoqian Wu, Cheng Huang. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 14.05.2025. – notice: COPYRIGHT 2025 Journal of Medical Internet Research – notice: 2025. This work is licensed under https://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. – notice: Dongmei Tan, Yi Huang, Ming Liu, Ziyu Li, Xiaoqian Wu, Cheng Huang. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 14.05.2025. 2025 |
| DBID | AAYXX CITATION CGR CUY CVF ECM EIF NPM 3V. 7QJ 7RV 7U3 7X7 7XB 8FI 8FJ 8FK ABUWG AFKRA ALSLI AZQEC BENPR BHHNA CCPQU CNYFK COVID DWQXO E3H F2A FYUFA GHDGH K9. KB0 M0S M1O NAPCQ PHGZM PHGZT PIMPY PKEHL PPXIY PQEST PQQKQ PQUKI PRINS PRQQA 7X8 5PM DOA |
| DOI | 10.2196/70733 |
| DatabaseName | CrossRef Medline MEDLINE MEDLINE (Ovid) MEDLINE MEDLINE PubMed ProQuest Central (Corporate) Applied Social Sciences Index & Abstracts (ASSIA) Nursing & Allied Health Database Social Services Abstracts Health & Medical Collection ProQuest Central (purchase pre-March 2016) Hospital Premium Collection Hospital Premium Collection (Alumni Edition) ProQuest Central (Alumni) (purchase pre-March 2016) ProQuest Central (Alumni) ProQuest Central UK/Ireland Social Science Premium Collection ProQuest Central Essentials ProQuest Central Sociological Abstracts ProQuest One Library & Information Science Collection Coronavirus Research Database ProQuest Central Library & Information Sciences Abstracts (LISA) Library & Information Science Abstracts (LISA) Proquest Health Research Premium Collection Health Research Premium Collection (Alumni) ProQuest Health & Medical Complete (Alumni) Nursing & Allied Health Database (Alumni Edition) ProQuest Health & Medical Collection Library Science Database Nursing & Allied Health Premium ProQuest Central Premium ProQuest One Academic (New) Publicly Available Content Database ProQuest One Academic Middle East (New) ProQuest One Health & Nursing ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central China ProQuest One Social Sciences MEDLINE - Academic PubMed Central (Full Participant titles) Open Access资源_DOAJ |
| DatabaseTitle | CrossRef MEDLINE Medline Complete MEDLINE with Full Text PubMed MEDLINE (Ovid) Publicly Available Content Database ProQuest One Academic Middle East (New) Library and Information Science Abstracts (LISA) ProQuest Central Essentials ProQuest Health & Medical Complete (Alumni) ProQuest Central (Alumni Edition) ProQuest One Community College ProQuest One Health & Nursing Applied Social Sciences Index and Abstracts (ASSIA) ProQuest Central China ProQuest Central ProQuest Library Science Health Research Premium Collection Health and Medicine Complete (Alumni Edition) ProQuest Central Korea Library & Information Science Collection Social Services Abstracts ProQuest Central (New) Social Science Premium Collection ProQuest One Social Sciences ProQuest One Academic Eastern Edition Coronavirus Research Database ProQuest Nursing & Allied Health Source ProQuest Hospital Collection Health Research Premium Collection (Alumni) ProQuest Hospital Collection (Alumni) Nursing & Allied Health Premium ProQuest Health & Medical Complete ProQuest One Academic UKI Edition Sociological Abstracts ProQuest Nursing & Allied Health Source (Alumni) ProQuest One Academic ProQuest One Academic (New) ProQuest Central (Alumni) MEDLINE - Academic |
| DatabaseTitleList | MEDLINE MEDLINE - Academic Publicly Available Content Database |
| Database_xml | – sequence: 1 dbid: DOA name: DOAJ Directory of Open Access Journals url: https://www.doaj.org/ sourceTypes: Open Website – sequence: 2 dbid: NPM name: PubMed url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 3 dbid: EIF name: MEDLINE url: https://proxy.k.utb.cz/login?url=https://www.webofscience.com/wos/medline/basic-search sourceTypes: Index Database – sequence: 4 dbid: BENPR name: Proquest Central url: https://www.proquest.com/central sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Medicine Library & Information Science Public Health |
| EISSN | 1438-8871 |
| ExternalDocumentID | oai_doaj_org_article_a23311cae3954166a7ce254b3b36bc65 PMC12120363 A839904394 40367512 10_2196_70733 |
| Genre | Journal Article |
| GeographicLocations | China |
| GeographicLocations_xml | – name: China |
| GroupedDBID | --- .4I .DC 29L 2WC 36B 53G 5GY 5VS 77K 7RV 7X7 8FI 8FJ AAFWJ AAKPC AAWTL AAYXX ABDBF ABIVO ABUWG ACGFO ADBBV AEGXH AENEX AFKRA AFPKN AIAGR ALIPV ALMA_UNASSIGNED_HOLDINGS ALSLI AOIJS BAWUL BCNDV BENPR CCPQU CITATION CNYFK CS3 DIK DU5 DWQXO E3Z EAP EBD EBS EJD ELW EMB EMOBN ESX F5P FRP FYUFA GROUPED_DOAJ GX1 HMCUK HYE IAO ICO IEA IHR INH ISN ITC KQ8 M1O M48 NAPCQ OK1 OVT P2P PGMZT PHGZM PHGZT PIMPY PQQKQ RNS RPM SJN SV3 TR2 UKHRP XSB CGR CUY CVF ECM EIF NPM PMFND 3V. 7QJ 7U3 7XB 8FK ACUHS AZQEC BHHNA COVID E3H F2A K9. PKEHL PPXIY PQEST PQUKI PRINS PRQQA 7X8 5PM PUEGO |
| ID | FETCH-LOGICAL-c460t-edc363990cf5f51c706c73b532403371f840426e8079bf4483eca5d8a4968dd03 |
| IEDL.DBID | DOA |
| ISSN | 1438-8871 1439-4456 |
| IngestDate | Wed Aug 27 00:53:02 EDT 2025 Thu Aug 21 18:37:20 EDT 2025 Fri Jul 11 17:53:22 EDT 2025 Fri Jul 25 09:19:04 EDT 2025 Tue Jun 17 21:56:13 EDT 2025 Tue May 27 03:55:11 EDT 2025 Mon Jun 02 02:22:21 EDT 2025 Tue Jul 01 04:49:59 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 5 |
| Keywords | latent Dirichlet allocation online health information large pretrained language models ChatGPT performance evaluation text similarity analysis artificial intelligence text generation information identification |
| Language | English |
| License | Dongmei Tan, Yi Huang, Ming Liu, Ziyu Li, Xiaoqian Wu, Cheng Huang. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 14.05.2025. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included. |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c460t-edc363990cf5f51c706c73b532403371f840426e8079bf4483eca5d8a4968dd03 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23 |
| ORCID | 0000-0001-6416-2445 0000-0001-7937-7166 0009-0002-5967-8396 0000-0002-3937-0557 0009-0001-6258-801X 0000-0003-0334-9075 |
| OpenAccessLink | https://doaj.org/article/a23311cae3954166a7ce254b3b36bc65 |
| PMID | 40367512 |
| PQID | 3222369417 |
| PQPubID | 2033121 |
| ParticipantIDs | doaj_primary_oai_doaj_org_article_a23311cae3954166a7ce254b3b36bc65 pubmedcentral_primary_oai_pubmedcentral_nih_gov_12120363 proquest_miscellaneous_3204329001 proquest_journals_3222369417 gale_infotracmisc_A839904394 gale_infotracacademiconefile_A839904394 pubmed_primary_40367512 crossref_primary_10_2196_70733 |
| ProviderPackageCode | CITATION AAYXX |
| PublicationCentury | 2000 |
| PublicationDate | 2025-05-14 |
| PublicationDateYYYYMMDD | 2025-05-14 |
| PublicationDate_xml | – month: 05 year: 2025 text: 2025-05-14 day: 14 |
| PublicationDecade | 2020 |
| PublicationPlace | Canada |
| PublicationPlace_xml | – name: Canada – name: Toronto – name: Toronto, Canada |
| PublicationTitle | Journal of medical Internet research |
| PublicationTitleAlternate | J Med Internet Res |
| PublicationYear | 2025 |
| Publisher | Journal of Medical Internet Research Gunther Eysenbach MD MPH, Associate Professor JMIR Publications |
| Publisher_xml | – name: Journal of Medical Internet Research – name: Gunther Eysenbach MD MPH, Associate Professor – name: JMIR Publications |
| References | ref13 ref35 ref12 ref34 ref15 ref37 ref14 ref36 ref31 Blei, DM (ref33) 2003; 3 ref30 ref11 ref10 ref32 ref2 ref1 ref17 ref39 ref16 ref38 ref19 ref18 ref24 ref23 ref26 ref25 ref20 ref42 ref41 ref22 ref21 ref43 Nan, X (ref5) 2021 ref28 ref27 ref29 ref8 ref7 ref9 ref4 ref3 ref6 ref40 |
| References_xml | – ident: ref39 doi: 10.1186/s12879-021-06670-y – ident: ref25 doi: 10.2196/19273 – ident: ref22 doi: 10.1001/jama.2018.16865 – ident: ref27 doi: 10.2196/52399 – ident: ref15 doi: 10.1007/s10439-023-03172-7 – ident: ref18 doi: 10.1007/s11431-020-1647-3 – volume: 3 start-page: 993 year: 2003 ident: ref33 publication-title: J Mach Learn Res – ident: ref11 doi: 10.1136/amiajnl-2012-001409 – ident: ref37 doi: 10.1016/j.ebiom.2023.104770 – ident: ref43 doi: 10.1197/jamia.m1687 – ident: ref12 doi: 10.1093/heapro/dag409 – ident: ref21 doi: 10.1001/jama.2024.21700 – ident: ref6 doi: 10.2753/mis0742-1222290408 – ident: ref17 doi: 10.3390/info13020083 – ident: ref30 doi: 10.1136/bmj.39489.470347.ad – ident: ref35 doi: 10.2196/17187 – ident: ref29 doi: 10.1145/3611651 – ident: ref32 doi: 10.1002/wics.101 – ident: ref31 doi: 10.1136/bmj.328.7454.1490 – ident: ref1 doi: 10.1016/j.chb.2014.10.050 – ident: ref10 doi: 10.1038/s41598-022-11488-y – ident: ref40 doi: 10.1001/jama.2024.6837 – ident: ref2 – ident: ref41 doi: 10.1016/j.patter.2024.100943 – ident: ref4 doi: 10.55529/jhtd.34.43.55 – ident: ref14 doi: 10.1109/jas.2023.123618 – ident: ref9 doi: 10.1016/j.socscimed.2019.112552 – ident: ref24 doi: 10.2196/19458 – ident: ref42 doi: 10.1080/28355245.2023.2263355 – ident: ref34 doi: 10.1016/j.tbench.2023.100105 – ident: ref28 – year: 2021 ident: ref5 publication-title: The Routledge Handbook of Health Communication – ident: ref7 doi: 10.1126/science.aao2998 – ident: ref20 doi: 10.3390/informatics11020013 – ident: ref16 doi: 10.3390/info10040150 – ident: ref19 doi: 10.1590/1806-9282.20230848 – ident: ref38 doi: 10.1093/bmb/ldab016 – ident: ref23 doi: 10.1126/science.aap9559 – ident: ref8 – ident: ref13 doi: 10.1016/j.socscimed.2007.01.012 – ident: ref36 doi: 10.2196/49771 – ident: ref3 doi: 10.2196/jmir.4018 – ident: ref26 doi: 10.1080/02604027.2019.1703158 |
| SSID | ssj0020491 |
| Score | 2.438793 |
| Snippet | Online health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or unverified... Background Online health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or... Background:Online health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or... BackgroundOnline health information is widely available, but a substantial portion of it is inaccurate or misleading, including exaggerated, incomplete, or... |
| SourceID | doaj pubmedcentral proquest gale pubmed crossref |
| SourceType | Open Website Open Access Repository Aggregation Database Index Database |
| StartPage | e70733 |
| SubjectTerms | Ability Accuracy Adaptability Artificial Intelligence Authenticity Cancer Chatbots Conspiracy Consumer Health Information Content analysis COVID-19 vaccines Credibility Cultural factors Data collection Datasets Discrepancies Evaluation False information Food Health information Health services Healthy food Humans Identification Infants Information Interfaces Internet Language Large language models Machine learning Maternal and infant welfare Medical decision making Misconceptions Misinformation Natural Language Processing Nutrition Online health care information services Original Paper Pandemics Public health Reliability Social networks Text categorization Topics Verification |
| SummonAdditionalLinks | – databaseName: Health & Medical Collection dbid: 7X7 link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1Ra9wwDBZbB2VQRnfb2mxt8WBsT6FJHNvJXko3VsrYlT6scG_GdpyuMJL27grtv6_k-K4Ng73ajoktyZZk6RPAJ1EVqraFT5UXPi2dLFMruU15ayshChfBdKZn8vSi_DkTs-hwW8SwytWZGA7qpnfkIz-kFwFOWZfq6PompapR9LoaS2g8hxcEXUZcrWaPBhdqv_kmbFG4MzLaoaIChaP7J8D0_3sYP7mNxpGST66ek214FXVGdjwQ-TU8890E9mPGAfvMYkoRbTGLsjqBzWl8NZ_A1uCbY0PK0Ru4HLJz2-iuY33LBsTROGI0YYgpYL8oYJydz30oKeEbbBj8nIyKqf1dfGXTqztsnoaC1AtG4Yn3b-Hi5Mfv76dpLLiQulJmyxTXyUljyVwrWpE7lUmnuBUBtI-rvEVrEG90X2VI3xYNO-6dEU1lylpWTZPxd7DR9Z3fBWYy26rcVMY3JV550jYuc5znwta2VMIkcLAig74ecDU02iNEJx3olMA3Is66k2CwQ0M_v9RRqrQpcMrcGc9rgZqlNMp5tHgtt1xaJ0UCX4i0moQV98eZmHOA_0iwV_q4otVScnACe6ORKGRu3L1iDh2FfKEfWTKBj-tu-pIC1zrf39IYwjysURlIYGfgpfWScFPRXMuLBKoRl43WPO7prv4ECPAcNQ56gn_____6AC8LqldMaLPlHmws57d-H5WopT0IkvIARxsbyw priority: 102 providerName: ProQuest |
| Title | Identification of Online Health Information Using Large Pretrained Language Models: Mixed Methods Study |
| URI | https://www.ncbi.nlm.nih.gov/pubmed/40367512 https://www.proquest.com/docview/3222369417 https://www.proquest.com/docview/3204329001 https://pubmed.ncbi.nlm.nih.gov/PMC12120363 https://doaj.org/article/a23311cae3954166a7ce254b3b36bc65 |
| Volume | 27 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV3da9RAEB-0QhGKaP2KtscKok-h2Wx2N_GtlZYiphaxcG_L7majBclJ7wr1v3dms3dc8MEXX_Kwm4TsfGRmdmd-A_BW1qVuXBlyHWTIK6-q3CnhctG7WsrSJzCd9kKdX1Wf5nK-1eqLcsJGeOCRcEe2FIJzb4NoJDoPymofMKhxwgnlvIropWjz1sFUCrXQ7-W7sEeJzihiR5paE04sTwTo__s3vGWHpjmSW0bn7DE8St4iOx6_8gncC8M-HKZaA_aOpWIiIi5LWroPu206L38K38c63D5tzLFFz0ZsUTaWH01eELMH2GdKDWeXNyE2jwgdDow7mozapv1cfmDt9R0Ot7H19JJRIuLvZ3B1dvrt43meWivkvlLFKsd1CfJNCt_LXnKvC-W1cDLC8wnNe4z70HaHukBO9hjCieCt7GpbNaruukI8h51hMYSXwGzhes1tbUNXoXFTrvOFR55J17hKS5vBbE1282tE0DAYeRBfTORLBifEjM0kAV7HARQDk8TA_EsMMnhPrDSklkgfb1N1AX4jAVyZ45pWS2XAGRxM7kR18tPptTCYpM5LQ8dRgkp-dQZvNtP0JKWoDWFxS_cQumGDZj-DF6PsbJaERMXAjJcZ1BOpmqx5OjNc_4hg3xx9Czpsf_U_qPQaHpbUv5jQZ6sD2Fnd3IZDdKpWbgb39VzP4MHJ6cXl11nUJry2_Msf9BMiXQ |
| linkProvider | Directory of Open Access Journals |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtR3ZbtQwcFSKVJAqBMvRQFuMxPEUNYnjOEFCqBzVlm4qHlpp31LbcUqlKim7W0F_im9kJnG2jZB466vtWLHnHs8B8Fqkkcx0ZH1phfVjk8S-Trj2eaVTISLjiunkh8n4OP42FdMV-NPnwlBYZc8TW0ZdNoZ85Dv0IsAp61J-vPjpU9coel3tW2h0aHFgr36hyTb_sP8F4fsmiva-Hn0e-66rgG_iJFj4tjScxHJgKlGJ0MggMZJr0Vam4zKs0ORBsWXTAA9RofXCrVGiTFWcJWlZBhz3vQN3UfAGZOzJ6bWBh9p2uAbrFF6NiL0jqSHiQN61bQH-Zf43pN8wMvOGqNt7CA-cjsp2O6R6BCu2HsGWy3Bgb5lLYSKQMscbRrCWu1f6Eax3vkDWpTg9htMuG7hy7kHWVKyrcOpWDDZsYxjYhALU2feZbVtY2BIHOr8qo-Zt5_P3LD_7jcN52wB7zigc8uoJHN8KKJ7Cat3UdgOYCnQlQ5UqW8YoYhNdmsBwHgqd6VgK5cF2D4bioqvjUaD9Q3AqWjh58ImAs5ykstvtQDM7LRwVFyrCLUOjLM8EarKJksaiha255ok2ifDgHYG2IOaA92OUy3HAf6QyW8VuSqelZGQPNgcrkajNcLpHjsIxlXlxTQIevFpO05cUKFfb5pLWUI3FDJUPD551uLQ8El4qmodh5EE6wLLBmYcz9dmPtuR4iBoOPfk___9_vYR746N8Ukz2Dw9ewP2IeiVTpdt4E1YXs0u7hQrcQm-3VMPg5LbJ9C8GlVcC |
| linkToPdf | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtR3batsw9NClEAZlbNnNW9tpsMuTiW1Zlj0Yo10b2rUJYazQN82S5a5Q7DZJ2fpr-7qdYytZzWBvfZVkYencj84F4I1II5npyPrSCuvHJol9nXDt81KnQkTGFdMZT5KDk_jLqThdg9_LXBgKq1zyxIZRF7UhH_mQXgQ4ZV3KYenCIqZ7o0-XVz51kKKX1mU7jRZFjuzNTzTf5h8P9xDWb6NotP_t84HvOgz4Jk6ChW8Lw0lEB6YUpQiNDBIjuRZNlTouwxLNHxRhNg3wQCVaMtyaXBRpHmdJWhQBx33vwbokq6gH67v7k-nXlbmHunfYhw0KtkY0H0pqj9iRfk2TgH9FwS1Z2I3TvCX4Rg_hgdNY2U6LYo9gzVYD2HL5DuwdcwlNBGDmOMUA-mP3Zj-AjdYzyNqEp8dw1uYGl85ZyOqStfVO3YrOhk1EAzumcHU2ndmmoYUtcKD1sjJq5XYx_8DG579weNy0w54zCo68eQIndwKMp9Cr6so-B5YHupRhnua2iFHgJrowgeE8FDrTsRS5B9tLMKjLtqqHQmuI4KQaOHmwS8BZTVIR7magnp0pR9Mqj3DL0OSWZwL12iSXxqK9rbnmiTaJ8OA9gVYRq8D7MbnLeMB_pKJbaiel01JqsgebnZVI4qY7vUQO5VjMXP0lCA9er6bpSwqbq2x9TWuo4mKGqogHz1pcWh0JLxWNxTDyIO1gWefM3Znq_EdTgDxEfYcCAF78_79eQR9JVB0fTo5ewv2IGidT2dt4E3qL2bXdQm1uobcd2TD4fteU-gfpMVyd |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Identification+of+Online+Health+Information+Using+Large+Pretrained+Language+Models%3A+Mixed+Methods+Study&rft.jtitle=Journal+of+medical+Internet+research&rft.au=Tan%2C+Dongmei&rft.au=Huang%2C+Yi&rft.au=Liu%2C+Ming&rft.au=Li%2C+Ziyu&rft.date=2025-05-14&rft.eissn=1438-8871&rft.volume=27&rft.spage=e70733&rft_id=info:doi/10.2196%2F70733&rft_id=info%3Apmid%2F40367512&rft.externalDocID=40367512 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1438-8871&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1438-8871&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1438-8871&client=summon |