Navigating the metric maze: a taxonomy of evaluation metrics for anomaly detection in time series
The field of time series anomaly detection is constantly advancing, with several methods available, making it a challenge to determine the most appropriate method for a specific domain. The evaluation of these methods is facilitated by the use of metrics, which vary widely in their properties. Despi...
Saved in:
| Published in | Data mining and knowledge discovery Vol. 38; no. 3; pp. 1027 - 1068 |
|---|---|
| Main Authors | , |
| Format | Journal Article |
| Language | English |
| Published |
New York
Springer US
01.05.2024
Springer Nature B.V |
| Subjects | |
| Online Access | Get full text |
Cover
Loading…
| Abstract | The field of time series anomaly detection is constantly advancing, with several methods available, making it a challenge to determine the most appropriate method for a specific domain. The evaluation of these methods is facilitated by the use of metrics, which vary widely in their properties. Despite the existence of new evaluation metrics, there is limited agreement on which metrics are best suited for specific scenarios and domains, and the most commonly used metrics have faced criticism in the literature. This paper provides a comprehensive overview of the metrics used for the evaluation of time series anomaly detection methods, and also defines a taxonomy of these based on how they are calculated. By defining a set of properties for evaluation metrics and a set of specific case studies and experiments, twenty metrics are analyzed and discussed in detail, highlighting the unique suitability of each for specific tasks. Through extensive experimentation and analysis, this paper argues that the choice of evaluation metric must be made with care, taking into account the specific requirements of the task at hand. |
|---|---|
| AbstractList | The field of time series anomaly detection is constantly advancing, with several methods available, making it a challenge to determine the most appropriate method for a specific domain. The evaluation of these methods is facilitated by the use of metrics, which vary widely in their properties. Despite the existence of new evaluation metrics, there is limited agreement on which metrics are best suited for specific scenarios and domains, and the most commonly used metrics have faced criticism in the literature. This paper provides a comprehensive overview of the metrics used for the evaluation of time series anomaly detection methods, and also defines a taxonomy of these based on how they are calculated. By defining a set of properties for evaluation metrics and a set of specific case studies and experiments, twenty metrics are analyzed and discussed in detail, highlighting the unique suitability of each for specific tasks. Through extensive experimentation and analysis, this paper argues that the choice of evaluation metric must be made with care, taking into account the specific requirements of the task at hand. |
| Author | Sørbø, Sondre Ruocco, Massimiliano |
| Author_xml | – sequence: 1 givenname: Sondre orcidid: 0000-0003-0673-5107 surname: Sørbø fullname: Sørbø, Sondre email: sondre.sorbo@sintef.no organization: Sintef Digital – sequence: 2 givenname: Massimiliano surname: Ruocco fullname: Ruocco, Massimiliano organization: Sintef Digital, Dept. of Computer Science, Norwegian University of Science and Technology |
| BookMark | eNp9kE1LAzEQhoNU0Fb_gKeA59XJxt1kvUnxC4peFLyFMZ2tke6mJmmx_npjWxA8eMqQeZ-Z4RmyQe97YuxEwJkAUOdRQC10AaUsABqtC73HDkWlZKGq-mWQa6kvikoLOGDDGN8BoColHDJ8wJWbYXL9jKc34h2l4Czv8IsuOfKEn7733Zr7ltMK58uc9P0uFXnrA8fcx_maTymR3XRdz5PriEcKjuIR229xHul4947Y88310_iumDze3o-vJoWVtUyFqKbwaiVdCF0r0JqwbbFpSlValKRIKEQrQFWtxcaKpsk_bVPaWk2tLmuSI3a6nbsI_mNJMZl3vwx9XmkkVFCpRkiRU-U2ZYOPMVBrFsF1GNZGgPlRabYqTVZpNiqNzpD-A1mXNiZSQDf_H5VbNOY9_YzC71X_UN8imIu3 |
| CitedBy_id | crossref_primary_10_1109_TKDE_2024_3466291 crossref_primary_10_1109_TPAMI_2024_3387317 crossref_primary_10_1109_ACCESS_2024_3392338 crossref_primary_10_2478_raft_2025_0016 crossref_primary_10_1016_j_cie_2024_110728 crossref_primary_10_1016_j_ifacol_2024_07_216 crossref_primary_10_1109_ACCESS_2024_3524908 crossref_primary_10_1016_j_ecoinf_2024_102922 |
| Cites_doi | 10.1109/TNNLS.2021.3136171 10.1109/BigData50022.2020.9378139 10.1109/TNNLS.2020.2980749 10.48550/arXiv.2210.01078 10.24963/ijcai.2019/616 10.1145/3357384.3358118 10.14778/3538598.3538602 10.1093/bib/bbr008 10.14778/3494124.3494142 10.1109/JIOT.2021.3100509 10.48550/arXiv.2202.07857 10.1016/j.oceaneng.2019.106129 10.1145/3534678.3539117 10.1109/ICMLA.2015.141 10.1145/3292500.3330680 10.1145/1143844.1143874 10.1109/IJCNN55064.2022.9891913 10.1145/3442381.3450023 10.1109/TITB.2005.863870 10.2478/ausi-2019-0008 10.1007/978-3-540-74767-3_14 10.1109/LRA.2018.2801475 10.1198/016214501753168136 10.1109/ICDE51399.2021.00228 10.1145/3442381.3450013 10.1007/s10661-020-8064-1 10.1155/2015/453214 10.1109/ACCESS.2020.2977892 10.48550/arXiv.2008.05788 10.1109/CVPRW53098.2021.00223 10.14778/3551793.3551830 10.1145/3394486.3403392 10.48550/arXiv.2201.07284 10.24963/ijcai.2019/837 10.1109/TNNLS.2022.3162949 10.1145/3485447.3511984 10.1109/TNNLS.2021.3105827 10.1109/TKDE.2021.3128667 10.1109/tkde.2021.3140058 10.1145/3178876.3185996 10.14778/3529337.3529354 10.1109/TNNLS.2020.3027736 10.1609/aaai.v35i5.16523 10.1007/978-981-15-1773-0_28 10.1016/j.neucom.2017.04.070, 10.1109/SSCI47803.2020.9308512 10.1145/3477314.3507024 10.1145/3292500.3330871 10.1371/journal.pone.0118432 10.1145/3219819.3219845 10.1109/TKDE.2021.3130234 10.1109/EPEC52095.2021.9621752 10.1109/TNNLS.2019.2935975 10.14778/3476249.3476307 10.1109/ICDE53745.2022.00116 10.23919/INM.2017.7987310 10.1145/3394486.3403378 10.1145/3447548.3467075 10.48550/arXiv.1201.0490 10.1145/3447548.3467174 10.3390/s20133738 10.1609/aaai.v36i7.20680 10.1111/J.1466-8238.2007.00358.X 10.1145/3292500.3330672 10.1109/SOCA.2019.00021 10.48550/arXiv.2212.11080 10.1109/ICDM50108.2020.00093 10.24963/ijcai.2019/378 10.1016/j.actaastro.2022.06.026, 10.1109/TII.2022.3164087 10.1609/aaai.v33i01.33011409 10.48550/arXiv.2004.00433 10.24963/ijcai.2020/173 10.1145/3534678.3539339 10.1088/1742-6596/1213/4/042050 10.48550/arXiv.2202.07586 10.1007/978-3-030-30490-4_56 10.48550/arXiv.2207.12201 10.3390/electronics11081213 10.1145/3534678.3539097 10.1109/TKDE.2022.3171562 10.1016/j.neucom.2021.03.062 10.1145/3447548.3467137 10.1016/j.measurement.2022.110791 10.48550/arXiv.2211.09224 10.1109/ACCESS.2021.3107975 |
| ContentType | Journal Article |
| Copyright | The Author(s) 2023 The Author(s) 2023. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| Copyright_xml | – notice: The Author(s) 2023 – notice: The Author(s) 2023. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| DBID | C6C AAYXX CITATION 3V. 7SC 7WY 7WZ 7XB 87Z 8AL 8AO 8FD 8FE 8FG 8FK 8FL 8G5 ABUWG AFKRA ARAPS AZQEC BENPR BEZIV BGLVJ CCPQU DWQXO FRNLG F~G GNUQQ GUQSH HCIFZ JQ2 K60 K6~ K7- L.- L7M L~C L~D M0C M0N M2O MBDVC P5Z P62 PHGZM PHGZT PKEHL PQBIZ PQBZA PQEST PQGLB PQQKQ PQUKI Q9U |
| DOI | 10.1007/s10618-023-00988-8 |
| DatabaseName | Springer Open Access Journals CrossRef ProQuest Central (Corporate) Computer and Information Systems Abstracts ABI/INFORM Collection ABI/INFORM Global (PDF only) ProQuest Central (purchase pre-March 2016) ABI/INFORM Collection Computing Database (Alumni Edition) ProQuest Pharma Collection Technology Research Database ProQuest SciTech Collection ProQuest Technology Collection ProQuest Central (Alumni) (purchase pre-March 2016) ABI/INFORM Collection (Alumni) ProQuest Research Library ProQuest Central (Alumni) ProQuest Central UK/Ireland Advanced Technologies & Aerospace Collection ProQuest Central Essentials ProQuest Databases Business Premium Collection Technology Collection ProQuest One Community College ProQuest Central Korea Business Premium Collection (Alumni) ABI/INFORM Global (Corporate) ProQuest Central Student ProQuest Research Library SciTech Premium Collection ProQuest Computer Science Collection ProQuest Business Collection (Alumni Edition) ProQuest Business Collection Computer Science Database ABI/INFORM Professional Advanced Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional ABI/INFORM Global Computing Database Research Library Research Library (Corporate) Advanced Technologies & Aerospace Database ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Premium ProQuest One Academic ProQuest One Academic Middle East (New) One Business ProQuest One Business (Alumni) ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Applied & Life Sciences ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central Basic |
| DatabaseTitle | CrossRef ABI/INFORM Global (Corporate) ProQuest Business Collection (Alumni Edition) ProQuest One Business Research Library Prep Computer Science Database ProQuest Central Student Technology Collection Technology Research Database Computer and Information Systems Abstracts – Academic ProQuest One Academic Middle East (New) ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Essentials ProQuest Computer Science Collection Computer and Information Systems Abstracts ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College Research Library (Alumni Edition) ProQuest Pharma Collection ABI/INFORM Complete ProQuest Central ABI/INFORM Professional Advanced ProQuest One Applied & Life Sciences ProQuest Central Korea ProQuest Research Library ProQuest Central (New) Advanced Technologies Database with Aerospace ABI/INFORM Complete (Alumni Edition) Advanced Technologies & Aerospace Collection Business Premium Collection ABI/INFORM Global ProQuest Computing ABI/INFORM Global (Alumni Edition) ProQuest Central Basic ProQuest Computing (Alumni Edition) ProQuest One Academic Eastern Edition ProQuest Technology Collection ProQuest SciTech Collection ProQuest Business Collection Computer and Information Systems Abstracts Professional Advanced Technologies & Aerospace Database ProQuest One Academic UKI Edition ProQuest One Business (Alumni) ProQuest One Academic ProQuest One Academic (New) ProQuest Central (Alumni) Business Premium Collection (Alumni) |
| DatabaseTitleList | ABI/INFORM Global (Corporate) CrossRef |
| Database_xml | – sequence: 1 dbid: C6C name: Springer Nature OA Free Journals url: http://www.springeropen.com/ sourceTypes: Publisher – sequence: 2 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Physics Computer Science |
| EISSN | 1573-756X |
| EndPage | 1068 |
| ExternalDocumentID | 10_1007_s10618_023_00988_8 |
| GrantInformation_xml | – fundername: Norges Forskningsråd grantid: 312062 funderid: http://dx.doi.org/10.13039/501100005416 – fundername: SINTEF |
| GroupedDBID | -59 -5G -BR -EM -Y2 -~C .4S .86 .DC .VR 06D 0R~ 0VY 199 1N0 1SB 203 29F 2J2 2JN 2JY 2KG 2KM 2LR 2P1 2VQ 2~H 30V 3V. 4.4 406 408 409 40D 40E 5GY 5VS 67Z 6NX 78A 7WY 8AO 8FE 8FG 8FL 8G5 8TC 8UJ 95- 95. 95~ 96X AABHQ AACDK AAHNG AAIAL AAJBT AAJKR AANZL AARHV AARTL AASML AATNV AATVU AAUYE AAWCG AAYIU AAYQN AAYTO AAYZH ABAKF ABBBX ABBXA ABDZT ABECU ABFTD ABFTV ABHLI ABHQN ABJNI ABJOX ABKCH ABKTR ABMNI ABMQK ABNWP ABQBU ABQSL ABSXP ABTEG ABTHY ABTKH ABTMW ABULA ABUWG ABWNU ABXPI ACAOD ACBXY ACDTI ACGFS ACHSB ACHXU ACKNC ACMDZ ACMLO ACOKC ACOMO ACPIV ACSNA ACZOJ ADHHG ADHIR ADINQ ADKNI ADKPE ADRFC ADTPH ADURQ ADYFF ADZKW AEBTG AEFQL AEGAL AEGNC AEJHL AEJRE AEKMD AEMSY AENEX AEOHA AEPYU AESKC AETLH AEVLU AEXYK AFBBN AFGCZ AFKRA AFLOW AFQWF AFWTZ AFZKB AGAYW AGDGC AGGDS AGJBK AGMZJ AGQEE AGQMX AGRTI AGWIL AGWZB AGYKE AHAVH AHBYD AHKAY AHSBF AHYZX AIAKS AIGIU AIIXL AILAN AITGF AJBLW AJRNO AJZVZ ALMA_UNASSIGNED_HOLDINGS ALWAN AMKLP AMXSW AMYLF AMYQR AOCGG ARAPS ARCSS ARMRJ ASPBG AVWKF AXYYD AYJHY AZFZN AZQEC B-. BA0 BDATZ BENPR BEZIV BGLVJ BGNMA BPHCQ BSONS C6C CAG CCPQU COF CS3 CSCUP DDRTE DL5 DNIVK DPUIP DU5 DWQXO EBLON EBS EDO EIOEI EJD ESBYG F5P FEDTE FERAY FFXSO FIGPU FINBP FNLPD FRNLG FRRFC FSGXE FWDCC GGCAI GGRSB GJIRD GNUQQ GNWQR GQ6 GQ7 GQ8 GROUPED_ABI_INFORM_COMPLETE GUQSH GXS H13 HCIFZ HF~ HG5 HG6 HMJXF HQYDN HRMNR HVGLF HZ~ I-F I09 IHE IJ- IKXTQ ITM IWAJR IXC IZIGR IZQ I~X J-C J0Z J9A JBSCW JCJTX JZLTJ K60 K6V K6~ K7- KDC KOV LAK LLZTM M0C M0N M2O M4Y MA- N2Q NB0 NPVJJ NQJWS NU0 O9- O93 O9J OAM OVD P2P P62 P9O PF0 PQBIZ PQBZA PQQKQ PROAC PT4 PT5 Q2X QOS R89 R9I RNI RNS ROL RPX RSV RZC RZE RZK S16 S1Z S27 S3B SAP SCO SDH SHX SISQX SJYHP SNE SNPRN SNX SOHCF SOJ SPISZ SRMVM SSLCW STPWE SZN T13 TEORI TSG TSK TSV TUC TUS U2A UG4 UOJIU UTJUX UZXMN VC2 VFIZW W23 W48 WK8 YLTOR Z45 Z7R Z7S Z7W Z7X Z7Y Z7Z Z81 Z83 Z88 ZMTXR AAPKM AAYXX ABBRH ABDBE ABFSG ACSTC ADHKG ADKFA AEZWR AFDZB AFHIU AFOHR AGQPQ AHPBZ AHWEU AIXLP AMVHM ATHPR AYFIA CITATION PHGZM PHGZT 7SC 7XB 8AL 8FD 8FK ABRTQ JQ2 L.- L7M L~C L~D MBDVC PKEHL PQEST PQGLB PQUKI Q9U |
| ID | FETCH-LOGICAL-c363t-15d0bc3e41867088eaffa99272ca3e7e17aac1075fca9c199e17f92c67dc826e3 |
| IEDL.DBID | BENPR |
| ISSN | 1384-5810 |
| IngestDate | Sat Aug 16 18:51:38 EDT 2025 Thu Apr 24 23:12:40 EDT 2025 Tue Jul 01 00:40:33 EDT 2025 Fri Feb 21 02:39:14 EST 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 3 |
| Keywords | Evaluation Taxonomy Anomaly detection Time series |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c363t-15d0bc3e41867088eaffa99272ca3e7e17aac1075fca9c199e17f92c67dc826e3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0003-0673-5107 |
| OpenAccessLink | https://doi.org/10.1007/s10618-023-00988-8 |
| PQID | 3050579131 |
| PQPubID | 43030 |
| PageCount | 42 |
| ParticipantIDs | proquest_journals_3050579131 crossref_primary_10_1007_s10618_023_00988_8 crossref_citationtrail_10_1007_s10618_023_00988_8 springer_journals_10_1007_s10618_023_00988_8 |
| ProviderPackageCode | CITATION AAYXX |
| PublicationCentury | 2000 |
| PublicationDate | 20240500 2024-05-00 20240501 |
| PublicationDateYYYYMMDD | 2024-05-01 |
| PublicationDate_xml | – month: 5 year: 2024 text: 20240500 |
| PublicationDecade | 2020 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York |
| PublicationTitle | Data mining and knowledge discovery |
| PublicationTitleAbbrev | Data Min Knowl Disc |
| PublicationYear | 2024 |
| Publisher | Springer US Springer Nature B.V |
| Publisher_xml | – name: Springer US – name: Springer Nature B.V |
| References | Feng C, Tian P (2021) Time series anomaly detection for cyber-physical systems via neural system identification and bayesian filtering. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery and data mining. Association for computing machinery, New York. KDD ’21, p 2858-2867, https://doi.org/10.1145/3447548.3467137 NiuZYuKWuXLstm-based vae-gan for time-series anomaly detectionSens Basel Switz202020373810.3390/s20133738 Audibert J, Michiardi P, Guyard F, et al. (2020) Usad: Unsupervised anomaly detection on multivariate time series. In: Proceedings of the 26th ACM SIGKDD International conference on knowledge discovery and data mining. Association for computing machinery, New York. KDD ’20, p 3395-3404, https://doi.org/10.1145/3394486.3403392 Scharwächter E, Müller E (2020) Statistical Evaluation of Anomaly Detectors for Sequences. In: 6th ACM SIGKDD workshop on mining and learning from time series (KDD MiLeTS 2020), https://doi.org/10.48550/arXiv.2008.05788 Zhang J, Wu D, Boulet B (2021) Time series anomaly detection for smart grids: A survey. 2021 IEEE electrical power and energy conference (EPEC) pp 125–130. https://doi.org/10.1109/EPEC52095.2021.9621752 Bashar MA, Nayak R (2020) Tanogan: Time series anomaly detection with generative adversarial networks. In: 2020 IEEE symposium series on computational intelligence, SSCI 2020, Canberra, December 1-4, 2020. IEEE, pp 1778–1785, https://doi.org/10.1109/SSCI47803.2020.9308512 KimGYLimSMEuomICA study on performance metrics for anomaly detection based on industrial control system operation dataElectronics2022118110821310.3390/electronics11081213 Lai K, Zha D, Xu J, et al. (2021) Revisiting time series outlier detection: Definitions and benchmarks. In: Vanschoren J, Yeung S (eds) Proceedings of the neural information processing systems track on datasets and benchmarks 1, NeurIPS datasets and benchmarks 2021, December 2021, virtual, https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ec5decca5ed3d6b8079e2e7e7bacc9f2-Abstract-round1.html Davis J, Goadrich M (2006) The relationship between precision-recall and ROC curves. In: Cohen WW, Moore AW (eds) Machine learning. Proceedings of the twenty-third international conference (ICML 2006). Pittsburgh, Pennsylvania, USA, June 25-29, 2006, ACM international conference proceeding series, vol 148. ACM, pp 233–240, https://doi.org/10.1145/1143844.1143874 Goswami M, Challu C, Callot L, et al. (2022) Unsupervised model selection for time-series anomaly detection. ArXiv abs/2210.01078. https://doi.org/10.48550/arXiv.2210.01078 Tatbul N, Lee TJ, Zdonik S, et al. (2018) Precision and recall for time series. In: Bengio S, Wallach HM, Larochelle H, et al. (eds) Advances in neural information processing systems 31: annual conference on neural information processing systems 2018. NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp 1924–1934, https://proceedings.neurips.cc/paper/2018/hash/8f468c873a32bb0619eaeb2050ba45d1-Abstract.html Xu H, Wang Y, Jian S, et al. (2022) Calibrated one-class classification for unsupervised time series anomaly detection. CoRR abs/2207.12201. https://doi.org/10.48550/arXiv.2207.12201 Geiger A, Liu D, Alnegheimish S, et al. (2020) Tadgan: Time series anomaly detection using generative adversarial networks. In: Wu X, Jermaine C, Xiong L, et al. (eds) 2020 IEEE international conference on big data (IEEE BigData 2020), Atlanta, GA, USA, December 10-13, 2020. IEEE, pp 33–43, https://doi.org/10.1109/BigData50022.2020.9378139 Hwang WS, Yun JH, Kim J, et al. (2022) "do you know existing accuracy metrics overrate time-series anomaly detections?". In: Proceedings of the 37th ACM/SIGAPP symposium on applied computing. Association for computing machinery. New York, SAC ’22, p 403-412, https://doi.org/10.1145/3477314.3507024 ChenZChenDYuanZLearning graph structures with transformer for multivariate time-series anomaly detection in IOTIEEE Internet Things J202199179918910.1109/JIOT.2021.3100509 Baireddy S, Desai SR, Mathieson JL, et al. (2021) Spacecraft time-series anomaly detection using transfer learning. In: 2021 IEEE/CVF Conference on computer vision and pattern recognition workshops (CVPRW), pp 1951–1960, https://doi.org/10.1109/CVPRW53098.2021.00223 Ahmed AH, Riegler MA, Hicks SA, et al. (2022) Rcad: Real-time collaborative anomaly detection system for mobile broadband networks. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. Association for computing machinery, New York. KDD ’22, p 2682-2691, https://doi.org/10.1145/3534678.3539097 Rewicki F, Denzler J, Niebling J (2022) Is it worth it? an experimental comparison of six deep- and classical machine learning methods for unsupervised anomaly detection in time series. ArXiv abs/2212.11080. https://doi.org/10.48550/arXiv.2212.11080 Huet A, Navarro JM, Rossi D (2022) Local evaluation of time series anomaly detection algorithms. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. Association for computing machinery. New York. KDD ’22, p 635-645, https://doi.org/10.1145/3534678.3539339 SivaraksHRatanamahatanaCRobust and accurate anomaly detection in ecg artifacts using time series motif discoveryComput Math Methods Med2015201545314330517010.1155/2015/453214 Doshi K, Abudalou S, Yilmaz Y (2022) Reward once, penalize once: Rectifying time series anomaly detection. In: International joint conference on neural networks, IJCNN 2022, Padua, July 18-23, 2022. IEEE, pp 1–8, https://doi.org/10.1109/IJCNN55064.2022.9891913 Nalepa J, Myller M, Andrzejewski J et al (2022) Evaluating algorithms for anomaly detection in satellite telemetry data. Acta Astronautica 198:689–701 https://doi.org/10.1016/j.actaastro.2022.06.026,www.sciencedirect.com/science/article/pii/S0094576522003162 Chuah MC, Fu F (2007) ECG anomaly detection via time series analysis. In: Thulasiraman P, He X, Xu TL, et al. (eds) Frontiers of high performance computing and networking ISPA 2007 workshops, ISPA 2007 international workshops SSDSN, UPWN, WISH, SGC, ParDMCom, HiPCoMB, and IST-AWSN Niagara Falls. August 28 - September 1, 2007, Proceedings, Lecture Notes in Computer Science, vol 4743. Springer, pp 123–135, https://doi.org/10.1007/978-3-540-74767-3_14 Schmidl S, Wenig P, Papenbrock T (2022) Anomaly detection in time series: a comprehensive evaluation. Proc VLDB Endow 15(9):1779-1797. https://doi.org/10.14778/3538598.3538602 Ahmad S, Lavin A, Purdy S et al (2017) Unsupervised real-time anomaly detection for streaming data. Neurocomputing 262:134–147 https://doi.org/10.1016/j.neucom.2017.04.070,www.sciencedirect.com/science/article/pii/S0925231217309864, online Real-Time Learning Strategies for Data Streams Zhang C, Song D, Chen Y, et al. (2018) A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. ArXiv abs/1811.08055. https://doi.org/10.1609/aaai.v33i01.33011409 Chen T, Liu X, Xia B, et al. (2020) Unsupervised anomaly detection of industrial robots using sliding-window convolutional variational autoencoder. IEEE Access 8:47,072–47,081. https://doi.org/10.1109/ACCESS.2020.2977892 SaitoTRehmsmeierMThe precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasetsPLoS ONE20151010.1371/journal.pone.0118432 LoboJMJiménez-ValverdeARealRAuc: a misleading measure of the performance of predictive distribution modelsGlob Ecol Biogeogr20081714515110.1111/J.1466-8238.2007.00358.X Campos D, Kieu T, Guo C, et al. (2021) Unsupervised time series outlier detection with diversity-driven convolutional ensembles. Proc VLDB Endow 15(3):611–623. https://doi.org/10.14778/3494124.3494142, http://www.vldb.org/pvldb/vol15/p611-chaves.pdf LiLYanJWenQLearning robust deep state space for unsupervised anomaly detection in contaminated time-seriesIEEE Trans Knowl Data Eng2022231110.1109/TKDE.2022.3171562 FengYLiuZChenJUnsupervised multimodal anomaly detection with missing sources for liquid rocket engineIEEE Trans Neural Netw Learn Syst2022911510.1109/TNNLS.2022.3162949 Li D, Chen D, Shi L, et al. (2019) Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In: International conference on artificial neural networks https://doi.org/10.1007/978-3-030-30490-4_56 LiuSZhouBDingQXTime series anomaly detection with adversarial reconstruction networksIEEE Trans Knowl Data Eng202210.1109/tkde.2021.3140058 Pedregosa F, Varoquaux G, Gramfort A, et al. (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830. https://doi.org/10.48550/arXiv.1201.0490 Dai E, Chen J (2022) Graph-augmented normalizing flows for anomaly detection of multiple time series. ArXiv abs/2202.07857. https://doi.org/10.48550/arXiv.2202.07857 DengLLianDHuangZGraph convolutional adversarial networks for spatiotemporal anomaly detectionIEEE Trans Neural Netw Learn Syst202233624162428444231610.1109/TNNLS.2021.3136171 WangYHanLLiuWStudy on wavelet neural network based anomaly detection in ocean observing data seriesOcean Eng201910.1016/j.oceaneng.2019.106129 Challu C, Jiang P, Wu YN, et al. (2022) Deep generative model with hierarchical latent factors for time series anomaly detection. In: International conference on artificial intelligence and statistics https://doi.org/10.48550/arXiv.2202.07586 ChenRShiGZhaoWA joint model for IT operation series prediction and anomaly detectionNeurocomputing202144813013910.1016/j.neucom.2021.03.062 Ma M, Zhang S, Chen J, et al. (2021) Jump-starting multivariate time series anomaly detection for online service systems. In: USENIX annual technical conference, https://www.usenix.org/conference/atc21/presentation/ma MamandipoorBMajdMSheikhalishahiSMonitoring and detecting faults in wastewater treatment plants using deep learningEnviron Monitor Assess202019211210.1007/s10661-020-8064-1 Paparrizos J, Boniol P, Palpanas T, et al. (2022a) Volume under the surface: A new accuracy evaluation measure for time-series a R Chen (988_CR14) 2021; 448 L Deng (988_CR24) 2022; 33 X Wang (988_CR84) 2022 Z Chen (988_CR16) 2022; 9 Z Niu (988_CR67) 2020; 20 988_CR26 988_CR25 988_CR69 EJ Keogh (988_CR48) 2006; 10 988_CR61 T Saito (988_CR75) 2015; 10 Y Wang (988_CR85) 2022; 18 988_CR20 988_CR64 988_CR22 988_CR66 988_CR21 988_CR65 988_CR68 988_CR23 B Du (988_CR27) 2021; 5 988_CR91 988_CR90 988_CR93 988_CR92 Y Wang (988_CR83) 2019 988_CR59 988_CR17 988_CR19 Z Chen (988_CR15) 2021; 9 988_CR18 988_CR51 Y Li (988_CR57) 2021; 23 988_CR95 988_CR94 988_CR53 988_CR11 988_CR55 988_CR10 988_CR54 988_CR13 S Liu (988_CR60) 2022 988_CR12 Y He (988_CR38) 2019; 4 G Kovács (988_CR52) 2019; 11 988_CR80 988_CR82 988_CR81 L Li (988_CR56) 2021; 32 988_CR4 988_CR3 988_CR2 988_CR1 L Li (988_CR58) 2022; 23 A Garg (988_CR32) 2022; 33 988_CR47 988_CR49 T Ergen (988_CR28) 2020; 31 988_CR40 DP Berrar (988_CR8) 2012; 13 988_CR42 988_CR86 Z He (988_CR39) 2020; 12 988_CR41 988_CR44 988_CR88 988_CR43 988_CR87 988_CR46 988_CR45 988_CR89 988_CR70 H Sivaraks (988_CR79) 2015; 2015 D Park (988_CR71) 2017; 3 B Mamandipoor (988_CR63) 2020; 192 Y Feng (988_CR29) 2022; 9 988_CR37 SG Baker (988_CR6) 2001; 96 988_CR7 988_CR36 JM Lobo (988_CR62) 2008; 17 988_CR5 988_CR9 988_CR73 988_CR72 988_CR31 GY Kim (988_CR50) 2022; 11 988_CR30 988_CR74 988_CR33 988_CR77 988_CR76 988_CR35 988_CR34 988_CR78 |
| References_xml | – reference: Flaborea A, Prenkaj B, Munjal B, et al. (2022) Are we certain it’s anomalous? ArXiv abs/2211.09224. https://doi.org/10.48550/arXiv.2211.09224 – reference: Kim S, Choi K, Choi H, et al. (2022b) Towards a rigorous evaluation of time-series anomaly detection. In: Thirty-sixth AAAI conference on artificial intelligence, AAAI 2022, Thirty-fourth conference on innovative applications of artificial intelligence, IAAI 2022, The twelveth symposium on educational advances in artificial intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022. AAAI Press, pp 7194–7201, https://ojs.aaai.org/index.php/AAAI/article/view/20680 – reference: Lai K, Zha D, Xu J, et al. (2021) Revisiting time series outlier detection: Definitions and benchmarks. In: Vanschoren J, Yeung S (eds) Proceedings of the neural information processing systems track on datasets and benchmarks 1, NeurIPS datasets and benchmarks 2021, December 2021, virtual, https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ec5decca5ed3d6b8079e2e7e7bacc9f2-Abstract-round1.html – reference: Chen X, Deng L, Huang F, et al. (2021b) DAEMON: unsupervised anomaly detection and interpretation for multivariate time series. In: 37th IEEE international conference on data engineering, ICDE 2021, Chania. April 19-22, 2021. IEEE, pp 2225–2230, https://doi.org/10.1109/ICDE51399.2021.00228, – reference: KimGYLimSMEuomICA study on performance metrics for anomaly detection based on industrial control system operation dataElectronics2022118110821310.3390/electronics11081213 – reference: Paparrizos J, Boniol P, Palpanas T, et al. (2022a) Volume under the surface: A new accuracy evaluation measure for time-series anomaly detection. Proc VLDB Endow 15:2774–2787. https://doi.org/10.14778/3551793.3551830 – reference: Shen L, Li Z, Kwok J (2020) Timeseries anomaly detection using temporal hierarchical one-class network. In: Larochelle H, Ranzato M, Hadsell R, et al. (eds) Advances in neural information processing systems, vol 33. curran associates, Inc., pp 13,016–13,026, https://proceedings.neurips.cc/paper/2020/file/97e401a02082021fd24957f852e0e475-Paper.pdf – reference: ChenRShiGZhaoWA joint model for IT operation series prediction and anomaly detectionNeurocomputing202144813013910.1016/j.neucom.2021.03.062 – reference: Abdulaal A, Liu Z, Lancewicki T (2021) Practical approach to asynchronous multivariate time series anomaly detection and localization. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery and data mining. Association for computing machinery, New York, NY, USA, KDD ’21, p 2485-2494, https://doi.org/10.1145/3447548.3467174, – reference: Challu C, Jiang P, Wu YN, et al. (2022) Deep generative model with hierarchical latent factors for time series anomaly detection. In: International conference on artificial intelligence and statistics https://doi.org/10.48550/arXiv.2202.07586 – reference: Feng C, Tian P (2021) Time series anomaly detection for cyber-physical systems via neural system identification and bayesian filtering. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery and data mining. Association for computing machinery, New York. KDD ’21, p 2858-2867, https://doi.org/10.1145/3447548.3467137, – reference: Paparrizos J, Kang Y, Boniol P, et al. (2022b) Tsb-uad: An end-to-end benchmark suite for univariate time-series anomaly detection. Proc VLDB Endow 15(8):1697-1711. https://doi.org/10.14778/3529337.3529354 – reference: SivaraksHRatanamahatanaCRobust and accurate anomaly detection in ecg artifacts using time series motif discoveryComput Math Methods Med2015201545314330517010.1155/2015/453214 – reference: Zhang CK, Li SZ, Zhang H, et al. (2020) Velc: A new variational autoencoder based model for time series anomaly detection. arXiv:1907.01702 – reference: Huet A, Navarro JM, Rossi D (2022) Local evaluation of time series anomaly detection algorithms. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. Association for computing machinery. New York. KDD ’22, p 635-645, https://doi.org/10.1145/3534678.3539339 – reference: Doshi K, Abudalou S, Yilmaz Y (2022) Reward once, penalize once: Rectifying time series anomaly detection. In: International joint conference on neural networks, IJCNN 2022, Padua, July 18-23, 2022. IEEE, pp 1–8, https://doi.org/10.1109/IJCNN55064.2022.9891913, – reference: GargAZhangWSamaranJAn evaluation of anomaly detection and diagnosis in multivariate time seriesIEEE Trans Neural Netw Learn Syst202233625082517444232310.1109/TNNLS.2021.3105827 – reference: Hundman K, Constantinou V, Laporte C, et al. (2018) Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In: Guo Y, Farooq F (eds) Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery and data mining, KDD 2018. London. August 19-23, 2018. ACM, pp 387–395, https://doi.org/10.1145/3219819.3219845 – reference: Wu R, Keogh EJ (2022) Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress (extended abstract). In: 2022 IEEE 38th international conference on data engineering (ICDE), pp 1479–1480, https://doi.org/10.1109/ICDE53745.2022.00116 – reference: Dai L, Lin T, Liu C, et al. (2021) Sdfvae: Static and dynamic factorized vae for anomaly detection of multivariate cdn kpis. In: Proceedings of the web conference 2021. Association for computing machinery, New York. WWW ’21, p 3076-3086, https://doi.org/10.1145/3442381.3450013, – reference: FengYLiuZChenJUnsupervised multimodal anomaly detection with missing sources for liquid rocket engineIEEE Trans Neural Netw Learn Syst2022911510.1109/TNNLS.2022.3162949 – reference: LiuSZhouBDingQXTime series anomaly detection with adversarial reconstruction networksIEEE Trans Knowl Data Eng202210.1109/tkde.2021.3140058 – reference: Huang T, Chen P, Li R (2022) A semi-supervised vae based active anomaly detection framework in multivariate time series for online systems. In: Proceedings of the ACM web conference 2022. Association for computing machinery. New York. WWW ’22, p 1797-1806, https://doi.org/10.1145/3485447.3511984, – reference: ParkDHoshiYKempCCA multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoderIEEE Robot Autom Lett201731544155110.1109/LRA.2018.2801475 – reference: Goodge A, Hooi B, Ng S, et al. (2020) Robustness of autoencoders for anomaly detection under adversarial impact. In: Bessiere C (ed) Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI 2020. ijcai.org, pp 1244–1250, https://doi.org/10.24963/ijcai.2020/173, – reference: Goswami M, Challu C, Callot L, et al. (2022) Unsupervised model selection for time-series anomaly detection. ArXiv abs/2210.01078. https://doi.org/10.48550/arXiv.2210.01078 – reference: DengLLianDHuangZGraph convolutional adversarial networks for spatiotemporal anomaly detectionIEEE Trans Neural Netw Learn Syst202233624162428444231610.1109/TNNLS.2021.3136171 – reference: LiLYanJWenQLearning robust deep state space for unsupervised anomaly detection in contaminated time-seriesIEEE Trans Knowl Data Eng2022231110.1109/TKDE.2022.3171562 – reference: Tuli S, Casale G, Jennings NR (2022) Tranad: deep transformer networks for anomaly detection in multivariate time series data. Proc VLDB Endow 15:1201–1214. https://doi.org/10.48550/arXiv.2201.07284 – reference: Hsieh RJ, Chou J, Ho CH (2019) Unsupervised online anomaly detection on multivariate sensing time series data for smart manufacturing. 2019 IEEE 12th conference on service-oriented computing and applications (SOCA) pp 90–97. https://doi.org/10.1109/SOCA.2019.00021 – reference: WangXPiDZhangXVariational transformer-based anomaly detection approach for multivariate time seriesMeasurement202210.1016/j.measurement.2022.110791 – reference: ChenZChenDYuanZLearning graph structures with transformer for multivariate time-series anomaly detection in IOTIEEE Internet Things J202199179918910.1109/JIOT.2021.3100509 – reference: Huang X, Lee J, Kwon YW, et al. (2020) Crowdquake: A networked system of low-cost sensors for earthquake detection via deep learning. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery and data mining https://doi.org/10.1145/3394486.3403378 – reference: Zhang J, Wu D, Boulet B (2021) Time series anomaly detection for smart grids: A survey. 2021 IEEE electrical power and energy conference (EPEC) pp 125–130. https://doi.org/10.1109/EPEC52095.2021.9621752 – reference: Ren H, Xu B, Wang Y, et al. (2019) Time-series anomaly detection service at microsoft. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining. Association for computing machinery. New York. KDD ’19, p 3009-3017, https://doi.org/10.1145/3292500.3330680, – reference: Ahmed AH, Riegler MA, Hicks SA, et al. (2022) Rcad: Real-time collaborative anomaly detection system for mobile broadband networks. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. Association for computing machinery, New York. KDD ’22, p 2682-2691, https://doi.org/10.1145/3534678.3539097, – reference: WangYDuXLuZImproved lstm-based time-series anomaly detection in rail transit operation environmentsIEEE Trans Indust Inform2022189027903610.1109/TII.2022.3164087 – reference: Scharwächter E, Müller E (2020) Statistical Evaluation of Anomaly Detectors for Sequences. In: 6th ACM SIGKDD workshop on mining and learning from time series (KDD MiLeTS 2020), https://doi.org/10.48550/arXiv.2008.05788 – reference: Choi K, Yi J, Park C, et al. (2021) Deep learning for anomaly detection in time-series data: Review, analysis, and guidelines. IEEE Access 9:120,043–120,065. https://doi.org/10.1109/ACCESS.2021.3107975 – reference: Hwang WS, Yun JH, Kim J, et al. (2022) "do you know existing accuracy metrics overrate time-series anomaly detections?". In: Proceedings of the 37th ACM/SIGAPP symposium on applied computing. Association for computing machinery. New York, SAC ’22, p 403-412, https://doi.org/10.1145/3477314.3507024, – reference: Bashar MA, Nayak R (2020) Tanogan: Time series anomaly detection with generative adversarial networks. In: 2020 IEEE symposium series on computational intelligence, SSCI 2020, Canberra, December 1-4, 2020. IEEE, pp 1778–1785, https://doi.org/10.1109/SSCI47803.2020.9308512 – reference: Kieu T, Yang B, Guo C, et al. (2019) Outlier detection for time series with recurrent autoencoder ensembles. In: International joint conference on artificial intelligence, https://doi.org/10.24963/ijcai.2019/378 – reference: Gensler A, Sick B (2014) Novel criteria to measure performance of time series segmentation techniques. In: Seidl T, Hassani M, Beecks C (eds) Proceedings of the 16th LWA Workshops: KDML, IR and FGWM, Aachen, Germany, September 8-10, 2014, CEUR workshop proceedings, vol 1226. CEUR-WS.org, pp 193–204, http://ceur-ws.org/Vol-1226/paper31.pdf – reference: Ma M, Zhang S, Chen J, et al. (2021) Jump-starting multivariate time series anomaly detection for online service systems. In: USENIX annual technical conference, https://www.usenix.org/conference/atc21/presentation/ma – reference: Meng H, Zhang Y, Li Y, et al. (2020) Spacecraft anomaly detection via transformer reconstruction error. In: Jing Z (ed) Proceedings of the international conference on aerospace system science and engineering 2019. Springer, Singapore, pp 351–362, https://doi.org/10.1007/978-981-15-1773-0_28 – reference: Tatbul N, Lee TJ, Zdonik S, et al. (2018) Precision and recall for time series. In: Bengio S, Wallach HM, Larochelle H, et al. (eds) Advances in neural information processing systems 31: annual conference on neural information processing systems 2018. NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp 1924–1934, https://proceedings.neurips.cc/paper/2018/hash/8f468c873a32bb0619eaeb2050ba45d1-Abstract.html – reference: HeYZhaoJTemporal convolutional networks for anomaly detection in time seriesJ Phys Conf Ser20194121310.1088/1742-6596/1213/4/042050 – reference: ChenZChenDZhangXLearning graph structures with transformer for multivariate time-series anomaly detection in iotIEEE Internet Things J202291291799189449618710.1109/JIOT.2021.3100509 – reference: Chuah MC, Fu F (2007) ECG anomaly detection via time series analysis. In: Thulasiraman P, He X, Xu TL, et al. (eds) Frontiers of high performance computing and networking ISPA 2007 workshops, ISPA 2007 international workshops SSDSN, UPWN, WISH, SGC, ParDMCom, HiPCoMB, and IST-AWSN Niagara Falls. August 28 - September 1, 2007, Proceedings, Lecture Notes in Computer Science, vol 4743. Springer, pp 123–135, https://doi.org/10.1007/978-3-540-74767-3_14, – reference: ErgenTKozatSSUnsupervised anomaly detection with LSTM neural networksIEEE Trans Neural Netw Learn Syst202031831273141413679710.1109/TNNLS.2019.2935975 – reference: Bhatia S, Jain A, Li P, et al. (2021) Mstream: Fast anomaly detection in multi-aspect streams. In: Proceedings of the web conference 2021. Association for computing machinery, New York. WWW ’21, p 3371-3382, https://doi.org/10.1145/3442381.3450023, – reference: MamandipoorBMajdMSheikhalishahiSMonitoring and detecting faults in wastewater treatment plants using deep learningEnviron Monitor Assess202019211210.1007/s10661-020-8064-1 – reference: Han S, Woo SS (2022) Learning sparse latent graph representations for anomaly detection in multivariate time series. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. Association for computing machinery, New York. KDD ’22, p 2977-2986, https://doi.org/10.1145/3534678.3539117, – reference: Li Z, Zhao Y, Han J, et al. (2021c) Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery and data mining. association for computing machinery, New York. KDD ’21, p 3220-3230, https://doi.org/10.1145/3447548.3467075, – reference: Zhou B, Liu S, Hooi B, et al. (2019) Beatgan: Anomalous rhythm detection using adversarially generated time series. In: International joint conference on artificial intelligence, https://doi.org/10.24963/ijcai.2019/616 – reference: Baireddy S, Desai SR, Mathieson JL, et al. (2021) Spacecraft time-series anomaly detection using transfer learning. In: 2021 IEEE/CVF Conference on computer vision and pattern recognition workshops (CVPRW), pp 1951–1960, https://doi.org/10.1109/CVPRW53098.2021.00223 – reference: Lavin A, Ahmad S (2015a) Evaluating real-time anomaly detection algorithms - the numenta anomaly benchmark. In: Li T, Kurgan LA, Palade V, et al. (eds) 14th IEEE international conference on machine learning and applications, ICMLA 2015, Miami. December 9-11, 2015. IEEE, pp 38–44, https://doi.org/10.1109/ICMLA.2015.141, – reference: BakerSGPinskyPFA proposed design and analysis for comparing digital and analog mammographyJ Am Stat Assoc20019645442142810.1198/016214501753168136 – reference: BerrarDPFlachPACaveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)Brief Bioinform2012131839710.1093/bib/bbr008 – reference: Xu H, Wang Y, Jian S, et al. (2022) Calibrated one-class classification for unsupervised time series anomaly detection. CoRR abs/2207.12201. https://doi.org/10.48550/arXiv.2207.12201, – reference: Rewicki F, Denzler J, Niebling J (2022) Is it worth it? an experimental comparison of six deep- and classical machine learning methods for unsupervised anomaly detection in time series. ArXiv abs/2212.11080. https://doi.org/10.48550/arXiv.2212.11080 – reference: Braei M, Wagner S (2020) Anomaly detection in univariate time-series: a survey on the state-of-the-art. CoRR abs/2004.00433. https://doi.org/10.48550/arXiv.2004.00433, arXiv:2004.00433 – reference: Campos D, Kieu T, Guo C, et al. (2021) Unsupervised time series outlier detection with diversity-driven convolutional ensembles. Proc VLDB Endow 15(3):611–623. https://doi.org/10.14778/3494124.3494142, http://www.vldb.org/pvldb/vol15/p611-chaves.pdf – reference: Dai E, Chen J (2022) Graph-augmented normalizing flows for anomaly detection of multiple time series. ArXiv abs/2202.07857. https://doi.org/10.48550/arXiv.2202.07857 – reference: Pang G, Shen C, van den Hengel A (2019) Deep anomaly detection with deviation networks. Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining https://doi.org/10.1145/3292500.3330871 – reference: WangYHanLLiuWStudy on wavelet neural network based anomaly detection in ocean observing data seriesOcean Eng201910.1016/j.oceaneng.2019.106129 – reference: LoboJMJiménez-ValverdeARealRAuc: a misleading measure of the performance of predictive distribution modelsGlob Ecol Biogeogr20081714515110.1111/J.1466-8238.2007.00358.X – reference: Hwang W, Yun J, Kim J, et al. (2019) Time-series aware precision and recall for anomaly detection: Considering variety of detection result and addressing ambiguous labeling. In: Zhu W, Tao D, Cheng X, et al. (eds) Proceedings of the 28th ACM international conference on information and knowledge management, CIKM 2019. Beijing, China, November 3-7, 2019. ACM, pp 2241–2244, https://doi.org/10.1145/3357384.3358118, – reference: KovácsGSebestyenGHanganAEvaluation metrics for anomaly detection algorithms in time-seriesActa Univ Sapientiae Inf20191111313010.2478/ausi-2019-0008 – reference: NiuZYuKWuXLstm-based vae-gan for time-series anomaly detectionSens Basel Switz202020373810.3390/s20133738 – reference: Pedregosa F, Varoquaux G, Gramfort A, et al. (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830. https://doi.org/10.48550/arXiv.1201.0490 – reference: Deng A, Hooi B (2021) Graph neural network-based anomaly detection in multivariate time series. In: Thirty-Fifth AAAI conference on artificial intelligence, AAAI 2021, Thirty-third conference on innovative applications of artificial intelligence, IAAI 2021, The eleventh symposium on educational advances in artificial intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. AAAI Press, pp 4027–4035, https://ojs.aaai.org/index.php/AAAI/article/view/16523 – reference: Geiger A, Liu D, Alnegheimish S, et al. (2020) Tadgan: Time series anomaly detection using generative adversarial networks. In: Wu X, Jermaine C, Xiong L, et al. (eds) 2020 IEEE international conference on big data (IEEE BigData 2020), Atlanta, GA, USA, December 10-13, 2020. IEEE, pp 33–43, https://doi.org/10.1109/BigData50022.2020.9378139, – reference: Su Y, Zhao Y, Niu C, et al. (2019) Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining. Association for computing machinery. New York. KDD ’19, p 2828-2837, https://doi.org/10.1145/3292500.3330672, – reference: Zhao H, Wang Y, Duan J, et al. (2020) Multivariate time-series anomaly detection via graph attention network. In: 2020 IEEE international conference on data mining (ICDM), pp 841–850, https://doi.org/10.1109/ICDM50108.2020.00093 – reference: Schmidl S, Wenig P, Papenbrock T (2022) Anomaly detection in time series: a comprehensive evaluation. Proc VLDB Endow 15(9):1779-1797. https://doi.org/10.14778/3538598.3538602, – reference: Buda TS, Assem H, Xu L (2017) ADE: an ensemble approach for early anomaly detection. In: 2017 IFIP/IEEE symposium on integrated network and service management (IM), Lisbon. May 8-12, 2017. IEEE, pp 442–448, https://doi.org/10.23919/INM.2017.7987310, – reference: Nalepa J, Myller M, Andrzejewski J et al (2022) Evaluating algorithms for anomaly detection in satellite telemetry data. Acta Astronautica 198:689–701 https://doi.org/10.1016/j.actaastro.2022.06.026,www.sciencedirect.com/science/article/pii/S0094576522003162 – reference: Wu R, Keogh EJ (2021) Ucr_anomalydatasets.pptx, supplemental material to the ucr anomaly archive. https://www.cs.ucr.edu/%7Eeamonn/time_series_data_2018/UCR_TimeSeriesAnomalyDatasets2021.zip, accessed: 2022-11-15 – reference: Zhang C, Song D, Chen Y, et al. (2018) A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. ArXiv abs/1811.08055. https://doi.org/10.1609/aaai.v33i01.33011409 – reference: HeZChenPLiXA spatiotemporal deep learning approach for unsupervised anomaly detection in cloud systemsIEEE Trans Neural Netw Learn Syst202012302773610.1109/TNNLS.2020.3027736 – reference: Xu H, Chen W, Zhao N, et al. (2018) Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In: Proceedings of the 2018 world wide web conference. International world wide web conferences steering committee, republic and canton of Geneva. CHE, WWW ’18, p 187-196, https://doi.org/10.1145/3178876.3185996, – reference: Jacob V, Song F, Stiegler A, et al. (2021) Exathlon: A benchmark for explainable anomaly detection over time series. Proc VLDB Endow 14(11), 2613–2626. https://doi.org/10.14778/3476249.3476307 – reference: Li D, Chen D, Shi L, et al. (2019) Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In: International conference on artificial neural networks https://doi.org/10.1007/978-3-030-30490-4_56 – reference: KeoghEJLinJFuAWCFinding unusual medical time-series subsequences: algorithms and applicationsIEEE Trans Inf Technol Biomed20061042943910.1109/TITB.2005.863870 – reference: Chen T, Liu X, Xia B, et al. (2020) Unsupervised anomaly detection of industrial robots using sliding-window convolutional variational autoencoder. IEEE Access 8:47,072–47,081. https://doi.org/10.1109/ACCESS.2020.2977892, – reference: LiLYanJWangHAnomaly detection of time series with smoothness-inducing sequential variational auto-encoderIEEE Trans Neural Netw Learn Syst20213231177119110.1109/TNNLS.2020.2980749 – reference: Zhang M, Li T, Shi H, et al. (2019) A decomposition approach for urban anomaly detection across spatiotemporal data. In: Kraus S (ed) Proceedings of the twenty-eighth international joint conference on artificial intelligence, IJCAI 2019, Macao. August 10-16, 2019. ijcai.org, pp 6043–6049, https://doi.org/10.24963/ijcai.2019/837, – reference: DuBSunXYeJGan-based anomaly detection for multivariate time series using polluted training setIEEE Trans Knowl Data Eng202151110.1109/TKDE.2021.3128667 – reference: Audibert J, Michiardi P, Guyard F, et al. (2020) Usad: Unsupervised anomaly detection on multivariate time series. In: Proceedings of the 26th ACM SIGKDD International conference on knowledge discovery and data mining. Association for computing machinery, New York. KDD ’20, p 3395-3404, https://doi.org/10.1145/3394486.3403392, – reference: SaitoTRehmsmeierMThe precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasetsPLoS ONE20151010.1371/journal.pone.0118432 – reference: Lavin A, Ahmad S (2015b) The numenta anomaly benchmark [White paper]. Redwood City, CA: Numenta, Available: https://github.com/numenta/NAB/wiki – reference: Ahmad S, Lavin A, Purdy S et al (2017) Unsupervised real-time anomaly detection for streaming data. Neurocomputing 262:134–147 https://doi.org/10.1016/j.neucom.2017.04.070,www.sciencedirect.com/science/article/pii/S0925231217309864, online Real-Time Learning Strategies for Data Streams – reference: LiYPengXZhangJDct-gan: dilated convolutional transformer-based gan for time series anomaly detectionIEEE Trans Knowl Data Eng2021231110.1109/TKDE.2021.3130234 – reference: Davis J, Goadrich M (2006) The relationship between precision-recall and ROC curves. In: Cohen WW, Moore AW (eds) Machine learning. Proceedings of the twenty-third international conference (ICML 2006). Pittsburgh, Pennsylvania, USA, June 25-29, 2006, ACM international conference proceeding series, vol 148. ACM, pp 233–240, https://doi.org/10.1145/1143844.1143874, – volume: 33 start-page: 2416 issue: 6 year: 2022 ident: 988_CR24 publication-title: IEEE Trans Neural Netw Learn Syst doi: 10.1109/TNNLS.2021.3136171 – ident: 988_CR33 doi: 10.1109/BigData50022.2020.9378139 – volume: 32 start-page: 1177 issue: 3 year: 2021 ident: 988_CR56 publication-title: IEEE Trans Neural Netw Learn Syst doi: 10.1109/TNNLS.2020.2980749 – ident: 988_CR36 doi: 10.48550/arXiv.2210.01078 – ident: 988_CR95 doi: 10.24963/ijcai.2019/616 – ident: 988_CR46 doi: 10.1145/3357384.3358118 – ident: 988_CR77 doi: 10.14778/3538598.3538602 – volume: 13 start-page: 83 issue: 1 year: 2012 ident: 988_CR8 publication-title: Brief Bioinform doi: 10.1093/bib/bbr008 – ident: 988_CR12 doi: 10.14778/3494124.3494142 – volume: 9 start-page: 9179 year: 2021 ident: 988_CR15 publication-title: IEEE Internet Things J doi: 10.1109/JIOT.2021.3100509 – ident: 988_CR21 doi: 10.48550/arXiv.2202.07857 – year: 2019 ident: 988_CR83 publication-title: Ocean Eng doi: 10.1016/j.oceaneng.2019.106129 – ident: 988_CR37 doi: 10.1145/3534678.3539117 – ident: 988_CR54 doi: 10.1109/ICMLA.2015.141 – ident: 988_CR73 doi: 10.1145/3292500.3330680 – ident: 988_CR23 doi: 10.1145/1143844.1143874 – ident: 988_CR26 doi: 10.1109/IJCNN55064.2022.9891913 – ident: 988_CR81 – ident: 988_CR9 doi: 10.1145/3442381.3450023 – volume: 10 start-page: 429 year: 2006 ident: 988_CR48 publication-title: IEEE Trans Inf Technol Biomed doi: 10.1109/TITB.2005.863870 – volume: 11 start-page: 113 year: 2019 ident: 988_CR52 publication-title: Acta Univ Sapientiae Inf doi: 10.2478/ausi-2019-0008 – ident: 988_CR20 doi: 10.1007/978-3-540-74767-3_14 – volume: 3 start-page: 1544 year: 2017 ident: 988_CR71 publication-title: IEEE Robot Autom Lett doi: 10.1109/LRA.2018.2801475 – volume: 96 start-page: 421 issue: 454 year: 2001 ident: 988_CR6 publication-title: J Am Stat Assoc doi: 10.1198/016214501753168136 – ident: 988_CR17 doi: 10.1109/ICDE51399.2021.00228 – ident: 988_CR22 doi: 10.1145/3442381.3450013 – volume: 192 start-page: 1 year: 2020 ident: 988_CR63 publication-title: Environ Monitor Assess doi: 10.1007/s10661-020-8064-1 – volume: 2015 start-page: 45314 year: 2015 ident: 988_CR79 publication-title: Comput Math Methods Med doi: 10.1155/2015/453214 – ident: 988_CR18 doi: 10.1109/ACCESS.2020.2977892 – volume: 9 start-page: 9179 issue: 12 year: 2022 ident: 988_CR16 publication-title: IEEE Internet Things J doi: 10.1109/JIOT.2021.3100509 – ident: 988_CR76 doi: 10.48550/arXiv.2008.05788 – ident: 988_CR5 doi: 10.1109/CVPRW53098.2021.00223 – ident: 988_CR69 doi: 10.14778/3551793.3551830 – ident: 988_CR4 doi: 10.1145/3394486.3403392 – ident: 988_CR82 doi: 10.48550/arXiv.2201.07284 – ident: 988_CR91 doi: 10.24963/ijcai.2019/837 – volume: 9 start-page: 1 year: 2022 ident: 988_CR29 publication-title: IEEE Trans Neural Netw Learn Syst doi: 10.1109/TNNLS.2022.3162949 – ident: 988_CR41 doi: 10.1145/3485447.3511984 – volume: 33 start-page: 2508 issue: 6 year: 2022 ident: 988_CR32 publication-title: IEEE Trans Neural Netw Learn Syst doi: 10.1109/TNNLS.2021.3105827 – ident: 988_CR34 – volume: 5 start-page: 1 year: 2021 ident: 988_CR27 publication-title: IEEE Trans Knowl Data Eng doi: 10.1109/TKDE.2021.3128667 – year: 2022 ident: 988_CR60 publication-title: IEEE Trans Knowl Data Eng doi: 10.1109/tkde.2021.3140058 – ident: 988_CR88 doi: 10.1145/3178876.3185996 – ident: 988_CR86 – ident: 988_CR55 – ident: 988_CR70 doi: 10.14778/3529337.3529354 – volume: 12 start-page: 3027736 year: 2020 ident: 988_CR39 publication-title: IEEE Trans Neural Netw Learn Syst doi: 10.1109/TNNLS.2020.3027736 – ident: 988_CR25 doi: 10.1609/aaai.v35i5.16523 – ident: 988_CR65 doi: 10.1007/978-981-15-1773-0_28 – ident: 988_CR2 doi: 10.1016/j.neucom.2017.04.070, – ident: 988_CR7 doi: 10.1109/SSCI47803.2020.9308512 – ident: 988_CR45 doi: 10.1145/3477314.3507024 – ident: 988_CR68 doi: 10.1145/3292500.3330871 – volume: 10 year: 2015 ident: 988_CR75 publication-title: PLoS ONE doi: 10.1371/journal.pone.0118432 – ident: 988_CR44 doi: 10.1145/3219819.3219845 – ident: 988_CR78 – volume: 23 start-page: 1 year: 2021 ident: 988_CR57 publication-title: IEEE Trans Knowl Data Eng doi: 10.1109/TKDE.2021.3130234 – ident: 988_CR93 doi: 10.1109/EPEC52095.2021.9621752 – volume: 31 start-page: 3127 issue: 8 year: 2020 ident: 988_CR28 publication-title: IEEE Trans Neural Netw Learn Syst doi: 10.1109/TNNLS.2019.2935975 – ident: 988_CR47 doi: 10.14778/3476249.3476307 – ident: 988_CR87 doi: 10.1109/ICDE53745.2022.00116 – ident: 988_CR11 doi: 10.23919/INM.2017.7987310 – ident: 988_CR42 doi: 10.1145/3394486.3403378 – ident: 988_CR61 doi: 10.1145/3447548.3467075 – ident: 988_CR72 doi: 10.48550/arXiv.1201.0490 – ident: 988_CR1 doi: 10.1145/3447548.3467174 – ident: 988_CR64 – volume: 20 start-page: 3738 year: 2020 ident: 988_CR67 publication-title: Sens Basel Switz doi: 10.3390/s20133738 – ident: 988_CR90 – ident: 988_CR51 doi: 10.1609/aaai.v36i7.20680 – volume: 17 start-page: 145 year: 2008 ident: 988_CR62 publication-title: Glob Ecol Biogeogr doi: 10.1111/J.1466-8238.2007.00358.X – ident: 988_CR80 doi: 10.1145/3292500.3330672 – ident: 988_CR40 doi: 10.1109/SOCA.2019.00021 – ident: 988_CR74 doi: 10.48550/arXiv.2212.11080 – ident: 988_CR94 doi: 10.1109/ICDM50108.2020.00093 – ident: 988_CR49 doi: 10.24963/ijcai.2019/378 – ident: 988_CR66 doi: 10.1016/j.actaastro.2022.06.026, – volume: 18 start-page: 9027 year: 2022 ident: 988_CR85 publication-title: IEEE Trans Indust Inform doi: 10.1109/TII.2022.3164087 – ident: 988_CR92 doi: 10.1609/aaai.v33i01.33011409 – ident: 988_CR10 doi: 10.48550/arXiv.2004.00433 – ident: 988_CR35 doi: 10.24963/ijcai.2020/173 – ident: 988_CR43 doi: 10.1145/3534678.3539339 – volume: 4 start-page: 1213 year: 2019 ident: 988_CR38 publication-title: J Phys Conf Ser doi: 10.1088/1742-6596/1213/4/042050 – ident: 988_CR13 doi: 10.48550/arXiv.2202.07586 – ident: 988_CR59 doi: 10.1007/978-3-030-30490-4_56 – ident: 988_CR89 doi: 10.48550/arXiv.2207.12201 – volume: 11 start-page: 1108213 issue: 8 year: 2022 ident: 988_CR50 publication-title: Electronics doi: 10.3390/electronics11081213 – ident: 988_CR3 doi: 10.1145/3534678.3539097 – volume: 23 start-page: 1 year: 2022 ident: 988_CR58 publication-title: IEEE Trans Knowl Data Eng doi: 10.1109/TKDE.2022.3171562 – volume: 448 start-page: 130 year: 2021 ident: 988_CR14 publication-title: Neurocomputing doi: 10.1016/j.neucom.2021.03.062 – ident: 988_CR30 doi: 10.1145/3447548.3467137 – year: 2022 ident: 988_CR84 publication-title: Measurement doi: 10.1016/j.measurement.2022.110791 – ident: 988_CR53 – ident: 988_CR31 doi: 10.48550/arXiv.2211.09224 – ident: 988_CR19 doi: 10.1109/ACCESS.2021.3107975 |
| SSID | ssj0005230 |
| Score | 2.4799705 |
| Snippet | The field of time series anomaly detection is constantly advancing, with several methods available, making it a challenge to determine the most appropriate... |
| SourceID | proquest crossref springer |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 1027 |
| SubjectTerms | Anomalies Artificial Intelligence Chemistry and Earth Sciences Computer Science Data Mining and Knowledge Discovery Information Storage and Retrieval Physics Statistics for Engineering Taxonomy Time series |
| SummonAdditionalLinks | – databaseName: Springer Open Access Journals dbid: C6C link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB60InjxURWrVXLwpoFms5tNvEmxiGBPFnpbstmsCO1W3FXUX-9kH64WFbzmdZg85hsy830Ap4jXDPcwOtFCSIoeL6FSJjENEl8azjViAlcofDsW1xP_ZhpMa5ocVwuz9H_vStwEkxQ9C3XUl5LKVVgLGA-dTMNQDL-kc_CqIlj6NJBsUBfI_LzGdyfUIsulz9DSx4y2YbMGh-Sy2s0dWLFZF7Ya4QVS38MurJd5mybfBT3WLyVJRnZPEMqRuRPIMmSu3-0F0aTQr2XRAlmkpOX1rkflBAEr0divZ28ksUWZlZWRh4w4xXniDqfN92AyurobXtNaNYEaLnhBWZAMYsOt76jq8A2xOk21Ul7oGc1taFmotcGgL0iNVoYphS2p8owIE4OxhuX70MkWmT0AgnOU9KUNY4w6jGdigcG0GnDDRIAPQdwD1pgxMjWluFO2mEUtGbIzfYSmj0rTR7IHZ59zHitCjT9H95vdierLlUfcqe-FinHWg_Nmx9ru31c7_N_wI9jwEMJU6Y196BRPz_YYIUgRn5Rn7wP8y890 priority: 102 providerName: Springer Nature |
| Title | Navigating the metric maze: a taxonomy of evaluation metrics for anomaly detection in time series |
| URI | https://link.springer.com/article/10.1007/s10618-023-00988-8 https://www.proquest.com/docview/3050579131 |
| Volume | 38 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1Lj9MwEB6x7WUvPHdFoVQ-cGOtrePEsbmgUvUhENUKbaXuKXIcZ4XUpmWTRcCvZ5w4BJDoJQfH9mFsz3xjz8wH8BrxmuEBeidaCEnR4mVUyiylURZKw7lGTOAShT-txHIdfthEG3_hVvqwylYn1oo62xt3R37JHeVarBhn7w5fqWONcq-rnkLjBPqogqXsQf_9bHX1-Y8gD97kCcuQRpKNfdqMT54TTFK0WdQV1ZRU_m2aOrz5zxNpbXnmj-Ghh4xk0qzxE3hgi6fwqKVjIP50PgO90t_qghnFLUFYR3aOLMuQnf5p3xJNKv29TmAg-5x0Nb59r5IgeCUa_-vtD5LZqo7QKsiXgjj2eeI2qi3PYD2fXU-X1DMoUMMFryiLsnFquA1d2TrUJ1bnuVYqiAOjuY0ti7U26ABGudHKMKWwJVeBEXFm0O-w_Bx6xb6wz4HgGCVDaeMUPRATmFSgY63G3DARoVJIB8Ba4SXGlxd3LBfbpCuM7ASeoMCTWuCJHMCb32MOTXGNo72H7Zok_qCVSbctBnDRrlP3-_-zvTg-20s4DRC-NKGNQ-hVd_f2FcKPKh3BiZwvRtCfLG4-zkZ-x2HrVEzxuw4mvwDR4tk0 |
| linkProvider | ProQuest |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1LT9wwEB4BPcAFKA-xPNo5wAks1nEedqUKVW2X5bUnkLgFx3GqSksW2PDqj-pvZJwHKZXgxjV-HMbjmW_imfkANgmvGeFRdKLDUDLyeCmTMk1YkPrSCKEJE7hC4ZNB2D_zD8-D8wn429TCuLTKxiaWhjodGfePfFc4yrVIccH3rq6ZY41yr6sNhUalFkf28Z5CtvHXgx90vlue1_t5-r3PalYBZkQoCsaDtJsYYX3Xyo3umNVZppXyIs9oYSPLI60NBUVBZrQyXCn6kinPhFFqCItbQftOwgdfkCd3lem9_X9SSkRVlSx9FkjerYt06lK9kEtGHpK5Fp6SyZeOsEW3_z3Iln6uNw-zNUDFb5VGfYQJmy_AXEP-gLUtWAQ90Hdle478FxKIxEtHzWXwUv-xX1BjoR_KcgkcZdh2FK9njZGgMmoa18NHTG1R5oPl-DtHx3WP7lrY8RKcvYtkl2EqH-V2BZDWKOlLGyUU7xjPJCGF8aorDA8DMkFJB3gjvNjUzcwdp8YwbtswO4HHJPC4FHgsO7D9vOaqauXx5uz15kzi-lqP41YJO7DTnFM7_Ppuq2_v9hmm-6cnx_HxweBoDWY8Ak5VUuU6TBU3t3aDgE-RfCq1DeHivdX7CRo2ESs |
| linkToPdf | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1LT9wwEB7RRap6KdCHuoXCHNoTWKzjPGykquK1glJWCBWJW-o4ToXEZmk3QOlP669jnDiEVio3rvHjMB7PfBPPfAPwnvCaEQFFJzqOJSOPlzMp84xFeSiNEJowgSsUPhzFeyfh59PodAb-tLUwLq2ytYm1oc4nxv0jXxeu5VqiuODrhU-LONoZfrr4wVwHKffS2rbTaFTkwN5cU_g2_bi_Q2f9IQiGu1-395jvMMCMiEXFeJQPMiNs6Gjd6L5ZXRRaqSAJjBY2sTzR2lCAFBVGK8OVoi-FCkyc5IZwuRW07xOYTVxU1IPZrd3R0fG9BBPR1CjLkEWSD3zJji_ci7lk5C-ZI_SUTP7tFjus-8_zbO31hvPw3MNV3Gz0awFmbPkC5tpWEOgtw0vQI31Vk3WU35EgJY5doy6DY_3bbqDGSv-qiydwUmDHL-5nTZGAM2oa1-c3mNuqzg4r8azE6mxs0V0SO30FJ48i29fQKyelfQNIa5QMpU0yin5MYLKYgno1EIbHERmkrA-8FV5qPLW567BxnnakzE7gKQk8rQWeyj6s3q25aIg9Hpy91J5J6i_5NO1Usg9r7Tl1w__f7e3Du63AU1Lt9Mv-6GARngWEopoMyyXoVT8v7TtCQVW27NUN4dtja_gtfCUWvQ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Navigating+the+metric+maze%3A+a+taxonomy+of+evaluation+metrics+for+anomaly+detection+in+time+series&rft.jtitle=Data+mining+and+knowledge+discovery&rft.au=S%C3%B8rb%C3%B8%2C+Sondre&rft.au=Ruocco%2C+Massimiliano&rft.date=2024-05-01&rft.pub=Springer+Nature+B.V&rft.issn=1384-5810&rft.eissn=1573-756X&rft.volume=38&rft.issue=3&rft.spage=1027&rft.epage=1068&rft_id=info:doi/10.1007%2Fs10618-023-00988-8&rft.externalDBID=HAS_PDF_LINK |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1384-5810&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1384-5810&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1384-5810&client=summon |