How Reliable Are Impact Factors for Ranking Water Resources Journals? An Analysis of the 70,878 Citation Records of the 2002 and 2021 Top 10 Journals for the Last Two Decades
Despite regularly being criticized, the Impact Factor (IF) remains a widely used journal analysis metric. In the Water Resources category of the Web of Science, a thorough investigation of the reliability of this metric is lacking. This study analyzed the citation records of the top 10 journals from...
Saved in:
Published in | Water resources research Vol. 59; no. 3 |
---|---|
Main Author | |
Format | Journal Article |
Language | English |
Published |
01.03.2023
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | Despite regularly being criticized, the Impact Factor (IF) remains a widely used journal analysis metric. In the Water Resources category of the Web of Science, a thorough investigation of the reliability of this metric is lacking. This study analyzed the citation records of the top 10 journals from 2002 through 2021 for the last two decades. Important to note, there are inconsistencies in the Web of Science database. Contradictory to many fields, few papers are uncited. However, the Simpson paradox applies. More specifically, on the one hand, the correlation between citations to papers 7 years post publication and 1 or 2 years post publication (which determine the IF) has increased, which supports this metric. On the other hand, many of the most highly cited papers at the time of the IF calculation are lowly cited 5 years afterward. Additionally, most currently very highly cited papers were relatively uncited when they were used in the IF calculation. This raises doubt about the usefulness of this metric. For a limited number of journals, papers from individual nations strongly influence the IF. Also, for a few journals, the IFs are strongly influenced by review papers, but leaving these out of the IF does change the top 3 and 10 of the rankings. The general conclusion is that IFs do contain some valuable information about the journals' citation impact, but that using them to rank journals is questionable practice.
Plain Language Summary
Impact Factors (IFs) continue to be very frequently used to evaluate researchers' publication records. They are calculated in a very simplistic manner, and are essentially an attempt to summarize a very complicated issue (publication quality) into one single metric. This paper shows that care must be taken when applying this practice. Not only does the data set contain errors, there are a number of other reasons why these IFs are not reliable indicators of a journal's quality.
Key Points
In many cases the calculation of the Impact Factor (IF) by the Web of Science is contradictory with their own data
Another number of issues have been identified which raise doubt on the IF as a measure of journal quality
A number of recommendations for a better use of the literature databases have been made |
---|---|
AbstractList | Despite regularly being criticized, the Impact Factor (IF) remains a widely used journal analysis metric. In the Water Resources category of the Web of Science, a thorough investigation of the reliability of this metric is lacking. This study analyzed the citation records of the top 10 journals from 2002 through 2021 for the last two decades. Important to note, there are inconsistencies in the Web of Science database. Contradictory to many fields, few papers are uncited. However, the Simpson paradox applies. More specifically, on the one hand, the correlation between citations to papers 7 years post publication and 1 or 2 years post publication (which determine the IF) has increased, which supports this metric. On the other hand, many of the most highly cited papers at the time of the IF calculation are lowly cited 5 years afterward. Additionally, most currently very highly cited papers were relatively uncited when they were used in the IF calculation. This raises doubt about the usefulness of this metric. For a limited number of journals, papers from individual nations strongly influence the IF. Also, for a few journals, the IFs are strongly influenced by review papers, but leaving these out of the IF does change the top 3 and 10 of the rankings. The general conclusion is that IFs do contain some valuable information about the journals' citation impact, but that using them to rank journals is questionable practice.
Plain Language Summary
Impact Factors (IFs) continue to be very frequently used to evaluate researchers' publication records. They are calculated in a very simplistic manner, and are essentially an attempt to summarize a very complicated issue (publication quality) into one single metric. This paper shows that care must be taken when applying this practice. Not only does the data set contain errors, there are a number of other reasons why these IFs are not reliable indicators of a journal's quality.
Key Points
In many cases the calculation of the Impact Factor (IF) by the Web of Science is contradictory with their own data
Another number of issues have been identified which raise doubt on the IF as a measure of journal quality
A number of recommendations for a better use of the literature databases have been made |
Author | Pauwels, Valentijn R. N. |
Author_xml | – sequence: 1 givenname: Valentijn R. N. orcidid: 0000-0002-1290-9313 surname: Pauwels fullname: Pauwels, Valentijn R. N. email: Valentijn.Pauwels@monash.edu organization: Monash University |
BookMark | eNpNUNtKw0AQXUTBennzA-YDjM7uZrPJk5R6qxSEUOljmCSzGo3Zko2U_pTfaLxQhGHODGfOGThHYr_zHQtxJvFCosouFSq1ylFrbdSemMgsjiObWb0vJoixjqTO7KE4CuEVUcYmsRPxee83kHPbUNkyTHuG-fuaqgFux-b7AM73kFP31nTPsKKBx42D_-grDvAwYkdtuIJpNxa129AE8A6GFwaL56lNYdYMNDS-G2WV7-sdrRAVUFePg5Kw9GuQuDP8-fp9taAwwHLj4ZorqjmciAM38nz6h8fi6fZmObuPFo9389l0EZFWRkbO1qVxzDWSslVJaFwqbZIY4oyZUuMMlzZO0solxtacZsYYclhWlcGyRH0s9K_vpml5W6z75p36bSGx-A66-B90scpnuUqMkvoLDVhzrw |
ContentType | Journal Article |
Copyright | 2023. The Authors. |
Copyright_xml | – notice: 2023. The Authors. |
DBID | 24P WIN |
DOI | 10.1029/2022WR033352 |
DatabaseName | Wiley Online Library Wiley Online Library |
DatabaseTitleList | |
Database_xml | – sequence: 1 dbid: 24P name: Wiley Online Library url: https://authorservices.wiley.com/open-science/open-access/browse-journals.html sourceTypes: Publisher |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Geography Economics |
EISSN | 1944-7973 |
EndPage | n/a |
ExternalDocumentID | WRCR26521 |
Genre | researchArticle |
GroupedDBID | -~X ..I .DC 05W 0R~ 123 1OB 1OC 24P 31~ 33P 3V. 50Y 5VS 6TJ 7WY 7XC 8-1 8CJ 8FE 8FG 8FH 8FL 8G5 8R4 8R5 8WZ A00 A6W AAESR AAHBH AAHHS AAIHA AAIKC AAMNW AANLZ AASGY AAXRX AAYJJ AAYOK AAZKR ABCUV ABJCF ABJNI ABPPZ ABTAH ABUWG ACAHQ ACBWZ ACCFJ ACCZN ACGFO ACGFS ACIWK ACKIV ACNCT ACPOU ACPRK ACXBN ACXQS ADBBV ADEOM ADKYN ADMGS ADOZA ADXAS ADZMN AEEZP AEIGN AENEX AEQDE AETEA AEUYR AFBPY AFGKR AFKRA AFPWT AFRAH AFZJQ AIDBO AIURR AIWBW AJBDE ALMA_UNASSIGNED_HOLDINGS ALUQN ALXUD AMYDB ASPBG ATCPS AVWKF AZFZN AZQEC AZVAB BDRZF BENPR BEZIV BFHJK BGLVJ BHPHI BKSAR BMXJE BPHCQ BRXPI CCPQU CS3 D0L D1J DCZOG DDYGU DPXWK DRFUL DRSTM DU5 DWQXO EBS EJD F5P FEDTE FRNLG G-S GNUQQ GODZA GROUPED_ABI_INFORM_COMPLETE GUQSH HCIFZ HVGLF HZ~ K60 K6~ L6V LATKE LEEKS LITHE LK5 LOXES LUTES LYRES M0C M2O M7R M7S MEWTI MSFUL MSSTM MVM MW2 MXFUL MXSTM MY~ O9- OHT OK1 P-X P2P P2W PALCI PATMY PCBAR PQBIZ PQBZA PQQKQ PROAC PTHSS PYCSY Q2X R.K RIWAO RJQFR ROL SAMSI SUPJJ TAE TN5 TWZ UQL VJK VOH WBKPD WIN WXSBR WYJ XOL XSW YHZ YV5 ZCG ZY4 ZZTAW ~02 ~KM ~OA ~~A |
ID | FETCH-LOGICAL-a3251-f7db5feed0a27cba05f817665ae9eea85f5eb7468cf657de89555af0bcc50bb03 |
IEDL.DBID | 24P |
ISSN | 0043-1397 |
IngestDate | Sat Aug 24 01:06:57 EDT 2024 |
IsDoiOpenAccess | true |
IsOpenAccess | true |
IsPeerReviewed | true |
IsScholarly | true |
Issue | 3 |
Language | English |
License | Attribution-NonCommercial-NoDerivs |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-a3251-f7db5feed0a27cba05f817665ae9eea85f5eb7468cf657de89555af0bcc50bb03 |
ORCID | 0000-0002-1290-9313 |
OpenAccessLink | https://onlinelibrary.wiley.com/doi/abs/10.1029%2F2022WR033352 |
PageCount | 18 |
ParticipantIDs | wiley_primary_10_1029_2022WR033352_WRCR26521 |
PublicationCentury | 2000 |
PublicationDate | March 2023 |
PublicationDateYYYYMMDD | 2023-03-01 |
PublicationDate_xml | – month: 03 year: 2023 text: March 2023 |
PublicationDecade | 2020 |
PublicationTitle | Water resources research |
PublicationYear | 2023 |
References | 2019; 7 2017; 2 2013; 1 2013; 2 2016; 108 2017; 27 2015; 30 2002; 54 2020; 15 2020; 106 1999; 43 2020; 56 2022; 42 2008; 4 2007; 11 2003; 31 2011; 7 2010; 84 2016; 13 2020; 19 2016; 55 2017; 53 2020; 5 2010; 20 2018; 391 2020; 2020 2020; 1 2000; 55 2013; 94 2019; 533 2020; 49 2011; 87 2020; 67 2014 2012; 6 2011; 28 2014; 8 1994; 30 |
References_xml | – volume: 2 issue: 1 year: 2013 article-title: Bibliometrics of systematic reviews: Analysis of citation rates and journal impact factors publication-title: Systematic Reviews – volume: 43 start-page: 971 issue: 10 year: 1999 end-page: 973 article-title: Citation analysis and journal impact factors—Is the tail wagging the dog? publication-title: Acta Anaesthesiologica Scandinavica – volume: 55 start-page: 1457 issue: 3 year: 2000 end-page: 1469 article-title: An analysis of finance journal impact factors author publication-title: The Journal of Finance – volume: 11 start-page: 1633 issue: 5 year: 2007 end-page: 1644 article-title: Updated world map of the Koppen‐Geiger climate classification publication-title: Hydrology and Earth System Sciences – volume: 30 start-page: 254 issue: 2 year: 2015 end-page: 259 article-title: Bibliometric indexes, databases and impact factors in cardiology publication-title: Brazilian Journal of Cardiovascular Surgery – volume: 1 start-page: 639 issue: 2 year: 2020 end-page: 663 article-title: Impact factor volatility due to a single paper: A comprehensive analysis publication-title: Quantitative Science Studies – volume: 54 start-page: 421 issue: 3 year: 2002 end-page: 434 article-title: Dynamic changes in the chance for citedness publication-title: Scientometrics – volume: 106 start-page: 1463 issue: 8 year: 2020 end-page: 1467 article-title: Bibliometric analysis of case report citations and their effect on the impact factor: How does publishing case reports impact journals? publication-title: Orthopaedics and Traumatology: Surgery and Research – volume: 13 start-page: 1391 issue: 11 year: 2016 end-page: 1396 article-title: The use of twitter by radiology journals: An analysis of twitter activity and impact factor publication-title: Journal of the American College of Radiology – volume: 53 start-page: 4533 issue: 6 year: 2017 end-page: 4541 article-title: The citation impact of hydrology journals publication-title: Water Resources Research – volume: 1 start-page: 1 issue: 1 year: 2013 end-page: 16 article-title: Citations and the h index of soil researchers and journals in the web of science, Scopus, and google scholar publication-title: PeerJ – volume: 55 start-page: 995 issue: 9 year: 2016 end-page: 999 article-title: Analysis of self‐citation and impact factor in dermatology journals publication-title: International Journal of Dermatology – volume: 7 start-page: 341 issue: 2 year: 2011 end-page: 354 article-title: Analysis of generalized impact factors and the indices of journals publication-title: Journal of Information Processing Systems – volume: 87 start-page: 551 issue: 3 year: 2011 end-page: 562 article-title: High‐impact papers presented in the subject category of water resources in the essential science indicators database of the institute for scientific information publication-title: Scientometrics – volume: 67 start-page: 90 issue: 2 year: 2020 end-page: 102 article-title: Mapping of top papers in the subject category of water resources based on the essential science indicators publication-title: Annals of Library and Information Studies – volume: 30 start-page: 65 issue: 1 year: 1994 end-page: 81 article-title: A bibliotheconomic analysis of the impact factors of scientific disciplines publication-title: Scientometrics – volume: 108 start-page: 1043 issue: 3 year: 2016 end-page: 1047 article-title: Does high impact factor successfully predict future citations? An analysis using Peirce's measure publication-title: Scientometrics – volume: 5 start-page: 2452 issue: 7 year: 2020 end-page: 2453 article-title: Impact factor, citescore, and citation analysis publication-title: ACS Energy Letters – volume: 4 start-page: 261 issue: 4 year: 2008 end-page: 274 article-title: Journal impact factors, h indices, and citation analyses in toxicology publication-title: Journal of Medical Toxicology: official journal of the American College of Medical Toxicology – volume: 42 start-page: 1141 issue: 6 year: 2022 end-page: 1142 article-title: The challenge of finding reviewers publication-title: Ophthalmic & Physiological Optics – volume: 8 start-page: 362 issue: 2 year: 2014 end-page: 368 article-title: Empirical modeling of the impact factor distribution publication-title: Journal of Informetrics – volume: 19 start-page: 1 year: 2020 end-page: 12 article-title: Publications on COVID‐19 in high impact factor journals: A bibliometric analysis publication-title: Universitas Psychologica – volume: 84 start-page: 65 issue: 1 year: 2010 end-page: 79 article-title: Comparative analysis between impact factor and h‐index for pharmacology and psychiatry journals publication-title: Scientometrics – volume: 533 year: 2019 article-title: The effect of “open access” on journal impact factors: A causal analysis of medical journals publication-title: Physica A: Statistical Mechanics and its Applications – volume: 20 start-page: 87 issue: 1 year: 2010 end-page: 96 article-title: Citation analysis and impact factor trends in the ergonomics journal list, 1975–2007 publication-title: Human Factors and Ergonomics in Manufacturing – volume: 7 start-page: 19775 year: 2019 end-page: 19781 article-title: Analysis of effects to journal impact factors based on citation networks generated via social computing publication-title: IEEE Access – volume: 28 start-page: 353 issue: 1–3 year: 2011 end-page: 365 article-title: Research articles published in water resources journals: A bibliometric analysis publication-title: Desalination and Water Treatment – volume: 94 start-page: 324 issue: 2 year: 2013 end-page: 331 article-title: Bibliometric analysis of literature in cerebrovascular and cardiovascular diseases rehabilitation: Growing numbers, reducing impact factor publication-title: Archives of Physical Medicine and Rehabilitation – start-page: 1576 year: 2014 end-page: 1585 article-title: Impact factor 2.0: Applying social network analysis to scientific impact assessment publication-title: Proceedings of the Annual Hawaii International Conference on System Sciences – volume: 2 year: 2017 article-title: Analysis of publications on journal impact factor over time publication-title: Frontiers in Research Metrics and Analytics – volume: 27 start-page: 97 issue: 2 year: 2017 end-page: 111 article-title: SCImago, Eigenfactor Score and H5 index journal rank indicator: Alternatives to the journal impact factor for water resources journals publication-title: Via Libre – volume: 391 start-page: 1033 issue: 9 year: 2018 end-page: 1035 article-title: Do review articles boost journal impact factors? A longitudinal analysis for five pharmacology journals publication-title: Naunyn‐Schmiedeberg's Archives of Pharmacology – volume: 49 start-page: 35 issue: 5 year: 2020 end-page: 57 article-title: Modeling and prediction of the impact factor of journals using open‐access databases publication-title: Austrian Journal of Statistics – volume: 6 start-page: 66 issue: 1 year: 2012 end-page: 79 article-title: Modeling the probabilistic distribution of the impact factor publication-title: Journal of Informetrics – volume: 84 start-page: 813 issue: 3 year: 2010 end-page: 820 article-title: A bibliometric analysis of the performance of water research publication-title: Scientometrics – volume: 2020 start-page: 1 year: 2020 end-page: 17 article-title: Modeling and simulation analysis of Journal impact factor dynamics based on submission and citation rules publication-title: Complexity – volume: 56 issue: 6 year: 2020 article-title: Invigorating hydrological research through journal publications publication-title: Water Resources Research – volume: 15 start-page: 1 issue: 5 year: 2020 end-page: 15 article-title: Comparison of researchers' impact indices publication-title: PLoS One – volume: 31 start-page: 14 issue: 1 year: 2003 end-page: 22 article-title: Citation analysis and journal impact factors in ophthalmology and vision science journals publication-title: Australian and New Zealand Journal of Ophthalmology |
SSID | ssj0014567 |
Score | 2.441854 |
Snippet | Despite regularly being criticized, the Impact Factor (IF) remains a widely used journal analysis metric. In the Water Resources category of the Web of... |
SourceID | wiley |
SourceType | Publisher |
SubjectTerms | impact factor web of science |
Title | How Reliable Are Impact Factors for Ranking Water Resources Journals? An Analysis of the 70,878 Citation Records of the 2002 and 2021 Top 10 Journals for the Last Two Decades |
URI | https://onlinelibrary.wiley.com/doi/abs/10.1029%2F2022WR033352 |
Volume | 59 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlZ09T8MwEIYtKAMsiE_xrRsYG-EmcexMqCpUBQFCUat2q-zUFlNSkaKKP8Vv5C5xqzKyOYqdSLEdPz7fvcfYbZwbJUPrcIpbFcRWqEBpLYMObnCdRIAwOdkhX9-SwSh-noiJN7hRLEyjD7E2uNHMqP_XNMG1qbzYAGlk4q49HGc8oqChbbaDZKNoVIfx-_oUAeFArk6YiXS84zu2v9ts_ZdL64Wlf8D2PRFCt-nCQ7ZliyO2uwoYrrDsE5V_fB-zn0G5BHIjpognbGPhqQ5zhH6TNweQQSHTdT4EGCNH4pU30FfgX1XdQ7eAlRoJlA4QAkHytpIKel6zG5p96fo2eVmALmZYCDswLOfQ4esH1m-lWi-6WsBwWcKDJb_76oSN-o_D3iDwKRcCHSHpBE7OjHC4bnIdytxoLpwiCUmhbWqtVsIJa2ScqNwlQs6sSoUQ2nGT54Ibw6NT1irKwp4xIIuSS9RMUqyAczLliJpRHmsTRQl35py1668-nTeyGtP6ODxMp5tdMx1nvSxMkDEu_lf9ku1RYvjGW-yKtRafX_Ya8WFhbuox8gukebtO |
link.rule.ids | 315,786,790,11589,27957,27958,46087,46511 |
linkProvider | Wiley-Blackwell |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3JTsMwELVYDuWCWMXOHDg2wk3i2D2hqlC1UBCqWpVbZKe2OCUVKUL8FN_ITOJW5cjNUexYijPx8yzvMXYTZ0bJ0Do0cauC2AoVKK1l0MIDrpMIIExGfsjnl6Q_iR_fxJvXOaVamJofYuVwI8uo_tdk4OSQ9mwDRJKJx_ZwOuIRVQ1tsm0S9SbLDOPXVRgB0YFchpgJ6vjMdxx_uz76LzCtdpbeHtv1kBA69Rrusw2bH7DGsmK4xLZXKn__PmQ__eILKI-YSp5wjIVBVecIvVo4BxCEwkhXgggwRSCJV95DX4KfqryDTg5LOhIoHCAKBMmbSiroetJuqA-mq9uUZgE6n2EjbMG4mEOLrx5YzUq9hrpcwPirgHtLifflEZv0HsbdfuA1FwIdIdQJnJwZ4XDj5DqUmdFcOEUckkLbtrVaCSeskXGiMpcIObOqLYTQjpssE9wYHh2zrbzI7QkDcim5RM0kFQs4J9scsWaUxdpEUcKdOWXN6q2n85pXI63i4WE7XV-adDrqjsIEQcbZ_7pfs0Z__DxMh4OXp3O2QyrxderYBdtafHzaS8QSC3NVfS-_wj2-rw |
linkToPdf | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3JTsMwELVYJOCCWMXOHDgS4SZx7J4QaqnKKlS1am-RndjilFSkCPFTfCMziVuVIzdH8cRSnInfbG8Yu4ozo2RoHaq4VUFshQqU1jJooYHrJAIIk5Ef8uU16Y_ix4mYeIcb1cI0_BALhxtpRv2_JgWf5s6TDRBHJlrt4XjAIyoaWmXrccIVGV9h_LaIIiA4kPMIMyEdn_iO8jfL0n9xaX2w9HbYtkeEcNds4S5bscUe25wXDFc49o3K37_32U-__AJKI6aKJ5Sx8FCXOUKv6ZsDiEFhoOt-CDBGHIlX3kFfgV-quoW7AuZsJFA6QBAIkl8rqaDjObuhsUsXtynLAnSR4yBswbCcQosvHlivSrOedTWD4VcJXUt599UBG_Xuh51-4FsuBDpCpBM4mRvh8NzkOpSZ0Vw4RRSSQtu2tVoJJ6yRcaIylwiZW9UWQmjHTZYJbgyPDtlaURb2iAF5lFyickm1As7JNkeoGWWxNlGUcGeO2XX91tNpQ6uR1uHwsJ0ub006HnQGYYIY4-R_0y_Zxlu3lz4_vD6dsi3qEd8kjp2xtdnHpz1HJDEzF_Xn8gvXTL3h |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+Reliable+Are+Impact+Factors+for+Ranking+Water+Resources+Journals%3F+An+Analysis+of+the+70%2C878+Citation+Records+of+the+2002+and+2021+Top+10+Journals+for+the+Last+Two+Decades&rft.jtitle=Water+resources+research&rft.au=Pauwels%2C+Valentijn+R.+N.&rft.date=2023-03-01&rft.issn=0043-1397&rft.eissn=1944-7973&rft.volume=59&rft.issue=3&rft.epage=n%2Fa&rft_id=info:doi/10.1029%2F2022WR033352&rft.externalDBID=10.1029%252F2022WR033352&rft.externalDocID=WRCR26521 |
thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0043-1397&client=summon |
thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0043-1397&client=summon |
thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0043-1397&client=summon |