The assessment of fundus image quality labeling reliability among graders with different backgrounds
For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality. Color fundus photographs were labeled using a Python-based...
Saved in:
| Published in | PloS one Vol. 17; no. 7; p. e0271156 |
|---|---|
| Main Authors | , , , |
| Format | Journal Article |
| Language | English |
| Published |
United States
Public Library of Science
26.07.2022
Public Library of Science (PLoS) |
| Subjects | |
| Online Access | Get full text |
Cover
Loading…
| Abstract | For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality.
Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis.
The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively.
Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. |
|---|---|
| AbstractList | For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality. Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis. The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively. Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. PurposeFor the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality.MethodsColor fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis.ResultsThe median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively.ConclusionImage grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality.PURPOSEFor the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality.Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis.METHODSColor fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis.The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively.RESULTSThe median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively.Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization.CONCLUSIONImage grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. Purpose For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality. Methods Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis. Results The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively. Conclusion Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality. Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis. The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen's weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively. Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. Purpose For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of graders with different backgrounds in the labeling of retinal fundus image quality. Methods Color fundus photographs were labeled using a Python-based tool using four image categories: excellent (E), good (G), adequate (A) and insufficient for grading (I). We enrolled 8 subjects (4 with and 4 without medical background, groups M and NM, respectively) to whom a tutorial was presented on image quality requirements. We randomly selected 200 images from a pool of 18,145 expert-labeled images (50/E, 50/G, 50/A, 50/I). The performance of the grading was timed and the agreement was assessed. An additional grading round was performed with 14 labels for a more objective analysis. Results The median time (interquartile range) for the labeling task with 4 categories was 987.8 sec (418.6) for all graders and 872.9 sec (621.0) vs. 1019.8 sec (479.5) in the M vs. NM groups, respectively. Cohen’s weighted kappa showed moderate agreement (0.564) when using four categories that increased to substantial (0.637) when using only three by merging the E and G groups. By the use of 14 labels, the weighted kappa values were 0.594 and 0.667 when assigning four or three categories, respectively. Conclusion Image grading with a Python-based tool seems to be a simple yet possibly efficient solution for the labeling of fundus images according to image quality that does not necessarily require medical background. Such grading can be subject to variability but could still effectively serve the robust identification of images with insufficient quality. This emphasizes the opportunity for the democratization of ML-applications among persons with both medical and non-medical background. However, simplicity of the grading system is key to successful categorization. |
| Audience | Academic |
| Author | Cabrera DeBuc, Delia Laurik-Feuerstein, Kornélia Lenke Sapahia, Rishav Somfai, Gábor Márk |
| AuthorAffiliation | 3 Department of Ophthalmology, Stadtspital Zürich, Zurich, Switzerland 5 Department of Ophthalmology, Semmelweis University, Budapest, Hungary 1 Department of Ophthalmology, Klinikum Ernst von Bergmann, Potsdam, Germany 4 Spross Research Institute, Zurich, Switzerland 2 Miller School of Medicine, Bascom Palmer Eye Institute, University of Miami, Miami, Florida, United States of America University of Warmia, POLAND |
| AuthorAffiliation_xml | – name: University of Warmia, POLAND – name: 3 Department of Ophthalmology, Stadtspital Zürich, Zurich, Switzerland – name: 2 Miller School of Medicine, Bascom Palmer Eye Institute, University of Miami, Miami, Florida, United States of America – name: 1 Department of Ophthalmology, Klinikum Ernst von Bergmann, Potsdam, Germany – name: 5 Department of Ophthalmology, Semmelweis University, Budapest, Hungary – name: 4 Spross Research Institute, Zurich, Switzerland |
| Author_xml | – sequence: 1 givenname: Kornélia Lenke surname: Laurik-Feuerstein fullname: Laurik-Feuerstein, Kornélia Lenke – sequence: 2 givenname: Rishav surname: Sapahia fullname: Sapahia, Rishav – sequence: 3 givenname: Delia surname: Cabrera DeBuc fullname: Cabrera DeBuc, Delia – sequence: 4 givenname: Gábor Márk orcidid: 0000-0001-6329-442X surname: Somfai fullname: Somfai, Gábor Márk |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/35881576$$D View this record in MEDLINE/PubMed |
| BookMark | eNqNk22L1DAQx4uceA_6DUQLguiLXZumTZt7IRyHDwsHB3r6NkzSaTdn2uwlrXrf3vS2K9vjEOmLlOlv_pP_dOY4Ouhsh1H0nCRLQgvy7toOrgOz3ITwMkkLQnL2KDoinKYLlib0YO_9MDr2_jpJcloy9iQ6pHlZkrxgR1F1tcYYvEfvW-z62NZxPXTV4GPdQoPxzQBG97exAYlGd03swgFS3wWhtSHSOKjQ-fiX7tdxpesa3agkQf1onA1i_mn0uAbj8dl0nkTfPn64Ov-8uLj8tDo_u1goxtN-kbIUCimLDDkWKqmyREmZS15iARzzDGRGCsKRZDUSSZTioAiUWSoZr2uq6En0cqu7MdaLqUFepIxnPEkZ5YFYbYnKwrXYuGDS3QoLWtwFrGsEuF4rg0KyOi-rtKhoSTIsC6hKXiEtgeV5QnketN5P1QbZYqWCaQdmJjr_0um1aOxPEf4KyTIaBN5MAs7eDOh70Wqv0Bjo0A7TvVnC6Vjr1T30YXcT1UAwoLvahrpqFBVn43zkWU5ZoJYPUOGpsNUqTFOtQ3yW8HaWEJgef_cNDN6L1dcv_89efp-zr_fYNYLp196aode283PwxX6n_7Z4N8YBON0CylnvHdZC6R5GnWBNG0ESMe7Mrmli3Bkx7UxIzu4l7_T_mfYH0HYcLA |
| CitedBy_id | crossref_primary_10_1007_s11357_024_01445_0 crossref_primary_10_1371_journal_pone_0289555 crossref_primary_10_1371_journal_pone_0292776 crossref_primary_10_1167_tvst_12_7_16 crossref_primary_10_1167_tvst_13_4_19 crossref_primary_10_1136_bmjophth_2023_001493 crossref_primary_10_2196_59914 |
| Cites_doi | 10.1001/jama.2016.17216 10.2147/OPTH.S235751 10.1167/iovs.12-10347 10.1167/iovs.05-1155 10.1016/S2589-7500(20)30240-5 10.1136/bjophthalmol-2018-313173 10.1016/j.ins.2019.06.011 10.1097/ICU.0000000000000678 10.1016/j.ophtha.2019.09.025 10.1016/j.ophtha.2019.12.015 10.1016/j.ophtha.2018.01.034 10.2196/jmir.3807 10.1016/j.preteyeres.2020.100900 10.1002/uog.5256 10.1177/193229680900300315 10.1038/s41591-018-0107-6 10.1186/s12886-016-0295-0 10.1371/journal.pone.0179310 |
| ContentType | Journal Article |
| Copyright | COPYRIGHT 2022 Public Library of Science 2022 Laurik-Feuerstein et al. This is an open access article distributed under the terms of the Creative Commons Attribution License: http://creativecommons.org/licenses/by/4.0/ (the “License”), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. 2022 Laurik-Feuerstein et al 2022 Laurik-Feuerstein et al |
| Copyright_xml | – notice: COPYRIGHT 2022 Public Library of Science – notice: 2022 Laurik-Feuerstein et al. This is an open access article distributed under the terms of the Creative Commons Attribution License: http://creativecommons.org/licenses/by/4.0/ (the “License”), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. – notice: 2022 Laurik-Feuerstein et al 2022 Laurik-Feuerstein et al |
| DBID | AAYXX CITATION CGR CUY CVF ECM EIF NPM IOV ISR 3V. 7QG 7QL 7QO 7RV 7SN 7SS 7T5 7TG 7TM 7U9 7X2 7X7 7XB 88E 8AO 8C1 8FD 8FE 8FG 8FH 8FI 8FJ 8FK ABJCF ABUWG AEUYN AFKRA ARAPS ATCPS AZQEC BBNVY BENPR BGLVJ BHPHI C1K CCPQU D1I DWQXO FR3 FYUFA GHDGH GNUQQ H94 HCIFZ K9. KB. KB0 KL. L6V LK8 M0K M0S M1P M7N M7P M7S NAPCQ P5Z P62 P64 PATMY PDBOC PHGZM PHGZT PIMPY PJZUB PKEHL PPXIY PQEST PQGLB PQQKQ PQUKI PRINS PTHSS PYCSY RC3 7X8 5PM DOA |
| DOI | 10.1371/journal.pone.0271156 |
| DatabaseName | CrossRef Medline MEDLINE MEDLINE (Ovid) MEDLINE MEDLINE PubMed Gale In Context: Opposing Viewpoints Gale In Context: Science ProQuest Central (Corporate) Animal Behavior Abstracts Bacteriology Abstracts (Microbiology B) Biotechnology Research Abstracts Nursing & Allied Health Database Ecology Abstracts Entomology Abstracts (Full archive) Immunology Abstracts Meteorological & Geoastrophysical Abstracts Nucleic Acids Abstracts Virology and AIDS Abstracts Agricultural Science Collection Health & Medical Collection ProQuest Central (purchase pre-March 2016) Medical Database (Alumni Edition) ProQuest Pharma Collection Public Health Database Technology Research Database ProQuest SciTech Collection ProQuest Technology Collection ProQuest Natural Science Collection Hospital Premium Collection Hospital Premium Collection (Alumni Edition) ProQuest Central (Alumni) (purchase pre-March 2016) Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest One Sustainability ProQuest Central UK/Ireland Advanced Technologies & Aerospace Collection Agricultural & Environmental Science Collection ProQuest Central Essentials Biological Science Collection ProQuest Central Technology Collection Natural Science Collection Environmental Sciences and Pollution Management ProQuest One ProQuest Materials Science Collection ProQuest Central Korea Engineering Research Database Health Research Premium Collection Health Research Premium Collection (Alumni) ProQuest Central Student AIDS and Cancer Research Abstracts SciTech Premium Collection ProQuest Health & Medical Complete (Alumni) Materials Science Database Nursing & Allied Health Database (Alumni Edition) Meteorological & Geoastrophysical Abstracts - Academic ProQuest Engineering Collection Biological Sciences Agriculture Science Database ProQuest Health & Medical Collection Medical Database Algology Mycology and Protozoology Abstracts (Microbiology C) Biological Science Database Engineering Database Nursing & Allied Health Premium Advanced Technologies & Aerospace Database ProQuest Advanced Technologies & Aerospace Collection Biotechnology and BioEngineering Abstracts Environmental Science Database Materials Science Collection ProQuest Central Premium ProQuest One Academic (New) Publicly Available Content Database ProQuest Health & Medical Research Collection ProQuest One Academic Middle East (New) ProQuest One Health & Nursing ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Applied & Life Sciences ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection Environmental Science Collection Genetics Abstracts MEDLINE - Academic PubMed Central (Full Participant titles) DOAJ Directory of Open Access Journals |
| DatabaseTitle | CrossRef MEDLINE Medline Complete MEDLINE with Full Text PubMed MEDLINE (Ovid) Agricultural Science Database Publicly Available Content Database ProQuest Central Student ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Essentials Nucleic Acids Abstracts SciTech Premium Collection ProQuest Central China Environmental Sciences and Pollution Management ProQuest One Applied & Life Sciences ProQuest One Sustainability Health Research Premium Collection Meteorological & Geoastrophysical Abstracts Natural Science Collection Health & Medical Research Collection Biological Science Collection ProQuest Central (New) ProQuest Medical Library (Alumni) Engineering Collection Advanced Technologies & Aerospace Collection Engineering Database Virology and AIDS Abstracts ProQuest Biological Science Collection ProQuest One Academic Eastern Edition Agricultural Science Collection ProQuest Hospital Collection ProQuest Technology Collection Health Research Premium Collection (Alumni) Biological Science Database Ecology Abstracts ProQuest Hospital Collection (Alumni) Biotechnology and BioEngineering Abstracts Environmental Science Collection Entomology Abstracts Nursing & Allied Health Premium ProQuest Health & Medical Complete ProQuest One Academic UKI Edition Environmental Science Database ProQuest Nursing & Allied Health Source (Alumni) Engineering Research Database ProQuest One Academic Meteorological & Geoastrophysical Abstracts - Academic ProQuest One Academic (New) Technology Collection Technology Research Database ProQuest One Academic Middle East (New) Materials Science Collection ProQuest Health & Medical Complete (Alumni) ProQuest Central (Alumni Edition) ProQuest One Community College ProQuest One Health & Nursing ProQuest Natural Science Collection ProQuest Pharma Collection ProQuest Central ProQuest Health & Medical Research Collection Genetics Abstracts ProQuest Engineering Collection Biotechnology Research Abstracts Health and Medicine Complete (Alumni Edition) ProQuest Central Korea Bacteriology Abstracts (Microbiology B) Algology Mycology and Protozoology Abstracts (Microbiology C) Agricultural & Environmental Science Collection AIDS and Cancer Research Abstracts Materials Science Database ProQuest Materials Science Collection ProQuest Public Health ProQuest Nursing & Allied Health Source ProQuest SciTech Collection Advanced Technologies & Aerospace Database ProQuest Medical Library Animal Behavior Abstracts Materials Science & Engineering Collection Immunology Abstracts ProQuest Central (Alumni) MEDLINE - Academic |
| DatabaseTitleList | MEDLINE - Academic MEDLINE Agricultural Science Database |
| Database_xml | – sequence: 1 dbid: DOA name: DOAJ Directory of Open Access Journals url: https://www.doaj.org/ sourceTypes: Open Website – sequence: 2 dbid: NPM name: PubMed url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 3 dbid: EIF name: MEDLINE url: https://proxy.k.utb.cz/login?url=https://www.webofscience.com/wos/medline/basic-search sourceTypes: Index Database – sequence: 4 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Sciences (General) |
| DocumentTitleAlternate | Image quality labeling among graders with different backgrounds |
| EISSN | 1932-6203 |
| ExternalDocumentID | 2694902639 oai_doaj_org_article_b6f58d27d3814e87ad89de38a6550395 PMC9321443 A711554536 35881576 10_1371_journal_pone_0271156 |
| Genre | Research Support, Non-U.S. Gov't Journal Article Research Support, N.I.H., Extramural |
| GeographicLocations | United States |
| GeographicLocations_xml | – name: United States |
| GrantInformation_xml | – fundername: NEI NIH HHS grantid: P30 EY014801 – fundername: ; – fundername: ; grantid: P30-EY014801 |
| GroupedDBID | --- 123 29O 2WC 53G 5VS 7RV 7X2 7X7 7XC 88E 8AO 8C1 8CJ 8FE 8FG 8FH 8FI 8FJ A8Z AAFWJ AAUCC AAWOE AAYXX ABDBF ABIVO ABJCF ABUWG ACGFO ACIHN ACIWK ACPRK ACUHS ADBBV AEAQA AENEX AEUYN AFKRA AFPKN AFRAH AHMBA ALIPV ALMA_UNASSIGNED_HOLDINGS AOIJS APEBS ARAPS ATCPS BAWUL BBNVY BCNDV BENPR BGLVJ BHPHI BKEYQ BPHCQ BVXVI BWKFM CCPQU CITATION CS3 D1I D1J D1K DIK DU5 E3Z EAP EAS EBD EMOBN ESX EX3 F5P FPL FYUFA GROUPED_DOAJ GX1 HCIFZ HH5 HMCUK HYE IAO IEA IGS IHR IHW INH INR IOV IPY ISE ISR ITC K6- KB. KQ8 L6V LK5 LK8 M0K M1P M48 M7P M7R M7S M~E NAPCQ O5R O5S OK1 OVT P2P P62 PATMY PDBOC PHGZM PHGZT PIMPY PQQKQ PROAC PSQYO PTHSS PV9 PYCSY RNS RPM RZL SV3 TR2 UKHRP WOQ WOW ~02 ~KM ADRAZ CGR CUY CVF ECM EIF IPNFZ NPM PJZUB PPXIY PQGLB RIG BBORY PMFND 3V. 7QG 7QL 7QO 7SN 7SS 7T5 7TG 7TM 7U9 7XB 8FD 8FK AZQEC C1K DWQXO FR3 GNUQQ H94 K9. KL. M7N P64 PKEHL PQEST PQUKI PRINS RC3 7X8 5PM PUEGO - 02 AAPBV ABPTK ADACO BBAFP KM |
| ID | FETCH-LOGICAL-c692t-262a7bb74e9e7c0d40cbb5b98e7a9e54ab41719e14fe1b1cc9ac1a842b69ff3c3 |
| IEDL.DBID | M48 |
| ISSN | 1932-6203 |
| IngestDate | Sun Jul 31 00:48:02 EDT 2022 Wed Aug 27 01:31:58 EDT 2025 Thu Aug 21 14:10:29 EDT 2025 Fri Jul 11 12:37:15 EDT 2025 Fri Jul 25 10:23:30 EDT 2025 Tue Jun 17 21:34:15 EDT 2025 Tue Jun 10 20:34:47 EDT 2025 Fri Jun 27 05:02:59 EDT 2025 Fri Jun 27 03:36:47 EDT 2025 Thu May 22 21:19:39 EDT 2025 Mon Jul 21 06:07:44 EDT 2025 Tue Jul 01 02:20:01 EDT 2025 Thu Apr 24 23:02:42 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 7 |
| Language | English |
| License | This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Creative Commons Attribution License |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c692t-262a7bb74e9e7c0d40cbb5b98e7a9e54ab41719e14fe1b1cc9ac1a842b69ff3c3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23 Competing Interests: The authors have declared that no competing interests exist. |
| ORCID | 0000-0001-6329-442X |
| OpenAccessLink | http://journals.scholarsportal.info/openUrl.xqy?doi=10.1371/journal.pone.0271156 |
| PMID | 35881576 |
| PQID | 2694902639 |
| PQPubID | 1436336 |
| PageCount | e0271156 |
| ParticipantIDs | plos_journals_2694902639 doaj_primary_oai_doaj_org_article_b6f58d27d3814e87ad89de38a6550395 pubmedcentral_primary_oai_pubmedcentral_nih_gov_9321443 proquest_miscellaneous_2694960935 proquest_journals_2694902639 gale_infotracmisc_A711554536 gale_infotracacademiconefile_A711554536 gale_incontextgauss_ISR_A711554536 gale_incontextgauss_IOV_A711554536 gale_healthsolutions_A711554536 pubmed_primary_35881576 crossref_citationtrail_10_1371_journal_pone_0271156 crossref_primary_10_1371_journal_pone_0271156 |
| ProviderPackageCode | CITATION AAYXX |
| PublicationCentury | 2000 |
| PublicationDate | 2022-07-26 |
| PublicationDateYYYYMMDD | 2022-07-26 |
| PublicationDate_xml | – month: 07 year: 2022 text: 2022-07-26 day: 26 |
| PublicationDecade | 2020 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States – name: San Francisco – name: San Francisco, CA USA |
| PublicationTitle | PloS one |
| PublicationTitleAlternate | PLoS One |
| PublicationYear | 2022 |
| Publisher | Public Library of Science Public Library of Science (PLoS) |
| Publisher_xml | – name: Public Library of Science – name: Public Library of Science (PLoS) |
| References | MA Zapata (pone.0271156.ref013) 2020; 14 A Kras (pone.0271156.ref003) 2020; 31 J Krause (pone.0271156.ref016) 2018; 125 R Thapa (pone.0271156.ref020) 2016; 16 J Cuadros (pone.0271156.ref012) 2009; 3 S Waite (pone.0271156.ref025) 2016 DSW Ting (pone.0271156.ref009) 2019; 103 J Loo (pone.0271156.ref010) 2020; 127 pone.0271156.ref006 pone.0271156.ref001 M McKenna (pone.0271156.ref022) 2018; 0 CJ Brady (pone.0271156.ref023) 2014; 16 DC DeBuc (pone.0271156.ref008) 2020; 12 E Trucco (pone.0271156.ref018) 2013; 54 CJ Flaxel (pone.0271156.ref007) 2020; 127 V Gulshan (pone.0271156.ref015) 2016; 316 AD Fleming (pone.0271156.ref014) 2006; 47 FMA Islam (pone.0271156.ref021) 2017; 12 SM Khan (pone.0271156.ref005) 2021; 3 J. W. Bartlett (pone.0271156.ref017) 2008; 31 T Panch (pone.0271156.ref004) 2020; 3 JPO Li (pone.0271156.ref019) 2021; 82 J De Fauw (pone.0271156.ref002) 2018; 24 T Li (pone.0271156.ref011) 2019; 501 CJ Brady (pone.0271156.ref024) 2021; 62 37796944 - PLoS One. 2023 Oct 5;18(10):e0292776 |
| References_xml | – ident: pone.0271156.ref001 – volume: 316 start-page: 2402 issue: 22 year: 2016 ident: pone.0271156.ref015 article-title: Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs publication-title: JAMA doi: 10.1001/jama.2016.17216 – volume: 14 start-page: 419 year: 2020 ident: pone.0271156.ref013 article-title: Artificial Intelligence to Identify Retinal Fundus Images, Quality Validation, Laterality Evaluation, Macular Degeneration, and Suspected Glaucoma publication-title: Clin Ophthalmol doi: 10.2147/OPTH.S235751 – volume: 54 start-page: 3546 issue: 5 year: 2013 ident: pone.0271156.ref018 article-title: Validating retinal fundus image analysis algorithms: issues and a proposal publication-title: Invest Ophthalmol Vis Sci doi: 10.1167/iovs.12-10347 – volume: 47 start-page: 1120 issue: 3 year: 2006 ident: pone.0271156.ref014 article-title: Automated assessment of diabetic retinal image quality based on clarity and field definition publication-title: Invest Ophth Vis Sci doi: 10.1167/iovs.05-1155 – volume: 3 issue: 87 year: 2020 ident: pone.0271156.ref004 article-title: “Yes, but will it work for my patients?” Driving clinically relevant research with benchmark datasets publication-title: npj Digit Med – volume: 3 start-page: e51 issue: 1 year: 2021 ident: pone.0271156.ref005 article-title: A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability publication-title: Lancet Digit Health doi: 10.1016/S2589-7500(20)30240-5 – year: 2016 ident: pone.0271156.ref025 article-title: Tired in the Reading Room: The Influence of Fatigue in Radiology publication-title: J Am Coll Radiol – volume: 0 start-page: 1 year: 2018 ident: pone.0271156.ref022 article-title: Accuracy of trained rural ophthalmologists versus non-medical image graders in the diagnosis of diabetic retinopathy in rural China publication-title: Br J Ophthalmol – volume: 12 start-page: 1 issue: 23 year: 2020 ident: pone.0271156.ref008 article-title: Artificial intelligence in the ophthalmic landscape publication-title: Nepal J Ophthalmol – volume: 103 start-page: 167 issue: 2 year: 2019 ident: pone.0271156.ref009 article-title: Artificial intelligence and deep learning in ophthalmology publication-title: Br J Ophthalmol doi: 10.1136/bjophthalmol-2018-313173 – volume: 501 start-page: 511 year: 2019 ident: pone.0271156.ref011 article-title: Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening publication-title: Inform Sciences doi: 10.1016/j.ins.2019.06.011 – volume: 31 start-page: 337 issue: 5 year: 2020 ident: pone.0271156.ref003 article-title: Accelerating ophthalmic artificial intelligence research: the role of an open access data repository publication-title: Curr Opin Ophthalmol doi: 10.1097/ICU.0000000000000678 – volume: 127 start-page: 66 issue: 1 year: 2020 ident: pone.0271156.ref007 article-title: Diabetic retinopathy preferredpractice pattern publication-title: Ophthalmology doi: 10.1016/j.ophtha.2019.09.025 – volume: 62 start-page: 1788 year: 2021 ident: pone.0271156.ref024 article-title: Crowdsourcing Can Match Field Grading Validity for Follicular Trachoma publication-title: IOVS – ident: pone.0271156.ref006 – volume: 127 start-page: 793 issue: 6 year: 2020 ident: pone.0271156.ref010 article-title: Beyond Performance Metrics: Automatic Deep Learning Retinal OCT Analysis Reproduces Clinical Trial Outcome publication-title: Ophthalmology doi: 10.1016/j.ophtha.2019.12.015 – volume: 125 start-page: 1264 issue: 8 year: 2018 ident: pone.0271156.ref016 article-title: Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy publication-title: Ophthalmology doi: 10.1016/j.ophtha.2018.01.034 – volume: 16 start-page: e233 issue: 10 year: 2014 ident: pone.0271156.ref023 article-title: Rapid grading of fundus photographs for diabetic retinopathy using crowdsourcing publication-title: J Med Internet Res doi: 10.2196/jmir.3807 – volume: 82 start-page: 100900 year: 2021 ident: pone.0271156.ref019 article-title: Digital technology, tele-medicine and artificial intelligence in ophthalmology: A global perspective publication-title: Prog Retin Eye Res doi: 10.1016/j.preteyeres.2020.100900 – volume: 31 start-page: 466 issue: 4 year: 2008 ident: pone.0271156.ref017 article-title: Reliability, Repeatability and Reproducibility: Analysis of Measurement Errors in Continuous Variables publication-title: Ultrasound in Obstetrics & Gynecology: The Official Journal of the International Society of Ultrasound in Obstetrics and Gynecology doi: 10.1002/uog.5256 – volume: 3 start-page: 509 issue: 3 year: 2009 ident: pone.0271156.ref012 article-title: EyePACS: an adaptable telemedicine system for diabetic retinopathy screening publication-title: J Diabetes Sci Technol doi: 10.1177/193229680900300315 – volume: 24 start-page: 1342 issue: 9 year: 2018 ident: pone.0271156.ref002 article-title: Clinically applicable deep learning for diagnosis and referral in retinal disease publication-title: Nat Med doi: 10.1038/s41591-018-0107-6 – volume: 16 start-page: 112 year: 2016 ident: pone.0271156.ref020 article-title: Intra- and inter-rater agreement between an ophthalmologist and mid-level ophthalmic personnel to diagnose retinal diseases based on fundus photographs at a primary eye center in Nepal: the Bhaktapur Retina Study publication-title: BMC Ophthalmol doi: 10.1186/s12886-016-0295-0 – volume: 12 start-page: e0179310 issue: 6 year: 2017 ident: pone.0271156.ref021 article-title: Accuracy and reliability of retinal photo grading for diabetic retinopathy: Remote graders from a developing country and standard retinal photo grader in Australia publication-title: PLoS One doi: 10.1371/journal.pone.0179310 – reference: 37796944 - PLoS One. 2023 Oct 5;18(10):e0292776 |
| SSID | ssj0053866 |
| Score | 2.447789 |
| Snippet | For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the performance of... Purpose For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the... PurposeFor the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the... Purpose For the training of machine learning (ML) algorithms, correctly labeled ground truth data are inevitable. In this pilot study, we assessed the... |
| SourceID | plos doaj pubmedcentral proquest gale pubmed crossref |
| SourceType | Open Website Open Access Repository Aggregation Database Index Database Enrichment Source |
| StartPage | e0271156 |
| SubjectTerms | Algorithms Artificial intelligence Automation Biology and Life Sciences Categories Color Color vision Computer and Information Sciences Datasets Diabetes Diabetic retinopathy Diagnostic Techniques, Ophthalmological Engineering and Technology Fundus Oculi Humans Image processing Image quality Labeling Labelling Labels Machine learning Medicine and Health Sciences Objective analysis Ophthalmology Performance assessment Photography - methods Pilot Projects Reliability analysis Reproducibility of Results Research and Analysis Methods Retina User interface |
| SummonAdditionalLinks | – databaseName: DOAJ Directory of Open Access Journals dbid: DOA link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1Lb9QwELbQnrggyqsLBQxCAg5pk9jx41gQVeEAElDUm-VXSkVJVpvdQ_8940eiBlUqB06RMpPHzsOe2cx8g9CrkllBae3AxbkvqK5MISDKL2xZu7Y0HoixQPYzOz6hn06b0yujvkJNWIIHToI7MKxthKu5g62FesG1E9J5IjSD2JrIiF4Ke96YTKU1GLyYsdwoR3h1kPWyv-o7vw-JGIRBbLYRRbz-aVVerC764bqQ8-_KyStb0dFddCfHkPgwvfsOuuW7e2gne-mA32Qo6bf3kQMrwHoC38R9i2Ejc9sBn_-GhQSnlspLDKYQ-9LxGg4JufsSxzlE-Gwda51x-MMWj-NUNtho-yt0hHRueIBOjj58f39c5LkKhWWy3hQ1qzU3hlMvPbelo6U1pjFSeK6lb6g2tOKV9BVtfWUqa6W2lRa0Nky2LbHkIVp0IMldhFuvG9PUFoKSFgIvK4mAG4UcTXuqqVgiMgpZ2Qw6HmZfXKj4JY1D8pFkpoJqVFbNEhXTVasEunED_7ugv4k3QGbHE2BIKhuSusmQluh50L5K_aeT46vD8AiIMwk85mXkCLAZXajLOdPbYVAfv_z4B6ZvX2dMrzNT24M4rM69EPCbAhzXjHNvxgnOb2fk3WCro1QGFRqTJeTVRMKVo_1eT34xkcNNQ61d5_tt5mHhA_kSPUrmPkmWNEJUkKIuEZ85wkz0c0p3_jOilsswEouSx_9DV0_Q7Tq0oZQcjHkPLTbrrX8KweHGPIvrwB8QF2Oa priority: 102 providerName: Directory of Open Access Journals – databaseName: Health & Medical Collection dbid: 7X7 link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1Lb9QwELZguXBBlFcDBQxCAg5pk9jx44QKoiocQAKK9hbZjrOtKMmy2T303zPjeANBFXCKFE-S3XnYM_bMN4Q8y4RTnBc1mLj0KTe5TRV4-anLirrJrIfBkCD7QRyf8Pfzch433PqYVrmdE8NEXXcO98gPsOJSQ8DA9KvljxS7RuHpamyhcZVcQ-gyTOmS8zHgAlsWIpbLMZkfROnsL7vW70M4Bs6QmCxHAbV_nJtny_Ouv8zx_DN_8rcF6egmuRE9SXo4iH6HXPHtLbITbbWnLyKg9MvbpAZdoGaE4KRdQ2E5qzc9PfsO0wkdCisvKChEqE6nK7gM-N0XNHQjootVyHimuG1Lt01V1tQa9w3rQtq6v0NOjt5-eXOcxu4KqRO6WKeFKIy0VnKvvXRZzTNnbWm18tJoX3JjeS5z7XPe-NzmzmnjcqN4YYVuGubYXTJrgZO7hDbelLYsHLgmDbhfTjMFL8JIzXhuuEoI2zK5chF6HDtgnFfhPE1CCDLwrELRVFE0CUnHp5YD9MY_6F-j_EZaBM4ON7rVoop2WFnRlKouZA2eCvdKmlrp2jNlBIRqTJcJeYzSr4Yq1NH8q0P8BHibDD7zNFAgeEaL2TkLs-n76t3Hr_9B9PnThOh5JGo6YIczsSIC_hOCck0o9yaUMAW4yfAu6uqWK331y1jgya3-Xj78ZBzGl2LGXeu7TaQReEyekHuDuo-cZaVSOQSqCZETQ5iwfjrSnp0G7HKNjbE4u__3n_WAXC-wzCSToKZ7ZLZebfxDcP7W9lGw8J9E7luh priority: 102 providerName: ProQuest |
| Title | The assessment of fundus image quality labeling reliability among graders with different backgrounds |
| URI | https://www.ncbi.nlm.nih.gov/pubmed/35881576 https://www.proquest.com/docview/2694902639 https://www.proquest.com/docview/2694960935 https://pubmed.ncbi.nlm.nih.gov/PMC9321443 https://doaj.org/article/b6f58d27d3814e87ad89de38a6550395 http://dx.doi.org/10.1371/journal.pone.0271156 |
| Volume | 17 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjR1db9Mw0Nq6F14Q42uFUQxCAh5S5cOJ7QeEtqllIDHQoKhvke04ZaIkpWkl-sJv585JI4LKx4sr9c6Ocr4738X3QcgTPzGCsTADEefWYyrQngAr3zN-mOW-tgB0AbIXyfmEvZnG0z2y7dnaELDa6dphP6nJcj78_m3zEgT-hevawIPtpOGiLOwQ3CwwcpJ9cgBnE0dRfcvaewWQbnd7iVaLl4R-1CTT_WmVzmHlavq3mru3mJfVLrP09-jKX46r8Q1yvbEz6UnNGIdkzxY3yWEjyRV91pSbfn6LZMApVLUFOmmZUzjssnVFr76CsqF12uWGAru43HW6hJ-6uveGul5FdLZ08dAUP-rSbcuVFdXKfMGskSKrbpPJePTx7Nxrei94JpHhyguTUHGtObPScuNnzDdax1oKy5W0MVMaqSttwHIb6MAYqUygBAt1IvM8MtEd0iuAkkeE5lbFOg4NGC45GGdGRgIWQj9OWaaY6JNoS-TUNIXJsT_GPHW3bRwclJpmKW5N2mxNn3jtrEVdmOMf-Ke4fy0ultV2f5TLWdpIaaqTPBZZyDOwY5gVXGVCZjYSKgFHLpJxnzzE3U_rHNVWOaQn-AiwRSN4zGOHgaU1Cozdmal1VaWv3336D6QPlx2kpw1SXgI5jGryJeCdsGRXB_O4gwkKwnTAR8irW6pUKSYvS_C9Iwkzt_y7G_yoBeOiGI9X2HLd4CR4id4nd2t2bykbxUIE4Mb2Ce8IQof0XUhx9dlVNpfYNotF9_7-QvfJtRCTUHwObHpMeqvl2j4A03ClB2SfTzmM4izAcfxqQA5ORxfvLwfuY8vAaQMcf4x-AlCLaUQ |
| linkProvider | Scholars Portal |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtR3LbtQw0CrLAS6I8upCoQaBgEPaPJzYPiBUHtUuLUWCFu0t2I6zVJRk2ewK7U_xjcwkTiCoAi49RYonzu68J54HIQ_9xAjGwgxEnFuPqUB7Arx8z_hhlvvawmKdIHuYjI7Zm0k8WSM_2loYTKtsdWKtqLPS4DfyHay4lBAwRPL57JuHU6PwdLUdodGwxb5dfYeQrXo2fgX0fRSGe6-PXo48N1XAM4kMF16YhIprzZmVlhs_Y77ROtZSWK6kjZnSLOCBtAHLbaADY6QygRIs1InM88hEsO8FchEMr48SxSddgAe6I0lceV7Egx3HDduzsrDbEP6B85X0zF89JaCzBYPZaVmd5ej-ma_5mwHcu0quOM-V7jastk7WbHGNrDvdUNEnroH10-skA96jqmv5ScucgvnMlhU9-QrqizaFnCsKDFhXw9M5XJp-4StaTz-i03mdYU3xMzFth7gsqFbmC9ahFFl1gxyfC95vkkEBmNwgNLcq1nFowBXKwd0zMhKwEUaGyjLFxJBELZJT41qd48SN07Q-v-MQ8jQ4S5E0qSPNkHjdU7Om1cc_4F8g_TpYbNRd3yjn09TJfaqTPBZZyDPwjJgVXGVCZjYSKoHQMJLxkGwh9dOm6rVTN-kuvgK82whe86CGwGYdBWYDTdWyqtLxu4__AfThfQ_osQPKS0CHUa4CA_4TNgHrQW72IEHlmN7yBvJqi5Uq_SWc8GTLv2cv3--WcVPM8CtsuXQwCR7LD8mtht07zEaxEAEExkPCe4LQQ31_pTj5XPdKlziIi0W3__6ztsil0dHbg_RgfLh_h1wOscTF58Cym2SwmC_tXXA8F_peLe2UfDpv9fITvH6aIg |
| linkToPdf | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtR1db9Mw0BpFQrwgxtcKgxkEAh6y5sOJ7QeEBqPaGBoI2NS3YDtOmRhJaVqh_jV-HXeJEwiagJc9RcpdnPa-fBffByEP_cQIxsIMVJxbj6lAewK8fM_4YZb72gKwTpA9TPaO2OtJPFkjP9paGEyrbG1ibaiz0uA38hFWXEoIGCI5yl1axLvd8fPZNw8nSOFJaztOoxGRA7v6DuFb9Wx_F3j9KAzHrz6-3PPchAHPJDJceGESKq41Z1ZabvyM-UbrWEthuZI2ZkqzgAfSBiy3gQ6MkcoESrBQJzLPIxPBuhfIRR7FAeoYn3TBHtiRJHGlehEPRk4ytmdlYbchFARHLOlthfXEgG5fGMxOy-osp_fP3M3fNsPxVXLFebF0pxG7dbJmi2tk3dmJij5xzayfXicZyCFVXftPWuYUttJsWdGTr2DKaFPUuaIgjHVlPJ3DpekdvqL1JCQ6ndfZ1hQ_GdN2oMuCamW-YE1KkVU3yNG50P0mGRRAyQ1Cc6tiHYcG3KIcXD8jIwELYZSoLFNMDEnUEjk1ru05Tt84TeuzPA7hT0OzFFmTOtYMidc9NWvafvwD_wXyr8PFpt31jXI-TZ0NSHWSxyILeQZeErOCq0zIzEZCJRAmRjIeki3kftpUwHamJ93BV4CnG8FrHtQY2LijQBWYqmVVpftvj_8D6cP7HtJjh5SXQA6jXDUG_CdsCNbD3OxhgvkxPfAGympLlSr9pajwZCu_Z4Pvd2BcFLP9ClsuHU6CR_RDcqsR946yUSxEAEHykPCeIvRI34cUJ5_rvukSh3Kx6Pbff9YWuQSGJX2zf3hwh1wOsdrF5yCxm2SwmC_tXfBBF_pereyUfDpv6_ITyJaeWA |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+assessment+of+fundus+image+quality+labeling+reliability+among+graders+with+different+backgrounds&rft.jtitle=PloS+one&rft.au=Laurik-Feuerstein%2C+Korn%C3%A9lia+Lenke&rft.au=Sapahia%2C+Rishav&rft.au=Cabrera+DeBuc%2C+Delia&rft.au=Somfai%2C+G%C3%A1bor+M%C3%A1rk&rft.date=2022-07-26&rft.pub=Public+Library+of+Science&rft.issn=1932-6203&rft.eissn=1932-6203&rft.volume=17&rft.issue=7&rft.spage=e0271156&rft_id=info:doi/10.1371%2Fjournal.pone.0271156&rft.externalDocID=A711554536 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1932-6203&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1932-6203&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1932-6203&client=summon |