A comprehensive empirical study on bug characteristics of deep learning frameworks
Deep Learning (DL) frameworks enable developers to build DNN models without learning the underlying algorithms and models. While some of these DL-based software systems have been deployed in safety-critical areas, such as self-driving cars and medical diagnostics, for DL frameworks, characterizing t...
Saved in:
| Published in | Information and software technology Vol. 151; p. 107004 |
|---|---|
| Main Authors | , , , |
| Format | Journal Article |
| Language | English |
| Published |
Elsevier B.V
01.11.2022
|
| Subjects | |
| Online Access | Get full text |
| ISSN | 0950-5849 1873-6025 |
| DOI | 10.1016/j.infsof.2022.107004 |
Cover
| Abstract | Deep Learning (DL) frameworks enable developers to build DNN models without learning the underlying algorithms and models. While some of these DL-based software systems have been deployed in safety-critical areas, such as self-driving cars and medical diagnostics, for DL frameworks, characterizing their bugs and thus helping researchers to design specific quality assurance techniques become desperately needed.
Our research aims to characterize bugs typical of DL frameworks at the source code level for an in-depth analysis of bug symptoms, root causes, and bug fixes. In this way, we hope to provide insights for researchers to design automatic quality assurance techniques, such as automatic repair techniques and fault location techniques, applicable to DL frameworks and DL-based software systems.
We started by summarizing the DL framework reference architecture and proposing the DL framework bug taxonomy. Then, we mined 1,127 DL framework bug reports from eight popular DL frameworks and labeled the bug types, root causes, and symptoms. Finally, we discussed the bug characteristics and explored how developers could possibly deal with these bugs.
Our main findings are: (i) DNN model building bugs and general type bugs accounted for one-third of the total defects. (ii) DNN model building bugs are more prone to algorithm logic constraints, internal API errors, and data/numerical errors. (iii) Fifteen bug-fixing patterns are summarized, providing reference for common DL framework bug repair and future research on the development of automatic DL framework bug detection tools.
By analyzing the bug-fixing changes, we characterize the occurrences, root causes, symptoms, and fixing of these bugs. The study results have provided researchers with insights into how to ensure DL framework quality and presented actionable suggestions for DL framework developers to improve their code quality. |
|---|---|
| AbstractList | Deep Learning (DL) frameworks enable developers to build DNN models without learning the underlying algorithms and models. While some of these DL-based software systems have been deployed in safety-critical areas, such as self-driving cars and medical diagnostics, for DL frameworks, characterizing their bugs and thus helping researchers to design specific quality assurance techniques become desperately needed.
Our research aims to characterize bugs typical of DL frameworks at the source code level for an in-depth analysis of bug symptoms, root causes, and bug fixes. In this way, we hope to provide insights for researchers to design automatic quality assurance techniques, such as automatic repair techniques and fault location techniques, applicable to DL frameworks and DL-based software systems.
We started by summarizing the DL framework reference architecture and proposing the DL framework bug taxonomy. Then, we mined 1,127 DL framework bug reports from eight popular DL frameworks and labeled the bug types, root causes, and symptoms. Finally, we discussed the bug characteristics and explored how developers could possibly deal with these bugs.
Our main findings are: (i) DNN model building bugs and general type bugs accounted for one-third of the total defects. (ii) DNN model building bugs are more prone to algorithm logic constraints, internal API errors, and data/numerical errors. (iii) Fifteen bug-fixing patterns are summarized, providing reference for common DL framework bug repair and future research on the development of automatic DL framework bug detection tools.
By analyzing the bug-fixing changes, we characterize the occurrences, root causes, symptoms, and fixing of these bugs. The study results have provided researchers with insights into how to ensure DL framework quality and presented actionable suggestions for DL framework developers to improve their code quality. |
| ArticleNumber | 107004 |
| Author | Feng, Yang He, Tianxing Xia, Zhilong Yang, Yilin |
| Author_xml | – sequence: 1 givenname: Yilin orcidid: 0000-0003-4746-8017 surname: Yang fullname: Yang, Yilin email: yilin.yang@smail.nju.edu.cn – sequence: 2 givenname: Tianxing surname: He fullname: He, Tianxing – sequence: 3 givenname: Zhilong surname: Xia fullname: Xia, Zhilong – sequence: 4 givenname: Yang surname: Feng fullname: Feng, Yang email: fengyang@nju.edu.cn |
| BookMark | eNqFkM1KAzEUhYNUsK2-gYu8wNT8zEw6LoRS_ANBEF2HTOamTZ1JhiSt9O2dMq5c6OrCge9w7jdDE-cdIHRNyYISWt7sFtaZ6M2CEcaGSBCSn6EpXQqelYQVEzQlVUGyYplXF2gW444QKggnU_S2wtp3fYAtuGgPgKHrbbBatTimfXPE3uF6v8F6q4LSCYKNyeqIvcENQI9bUMFZt8EmqA6-fPiMl-jcqDbC1c-do4-H-_f1U_by-vi8Xr1kmpMyZZAzMewtFGMUqDE1q0WlhplCV9zUQuU6b8paFKLIyyUFA6qqBOe0qVVNFeFzdDv26uBjDGCktkkl610KyraSEnmyI3dytCNPduRoZ4DzX3AfbKfC8T_sbsRgeOxgIcioLTgNjQ2gk2y8_bvgG36hhDE |
| CitedBy_id | crossref_primary_10_1109_TSE_2024_3461657 crossref_primary_10_1016_j_jss_2023_111681 crossref_primary_10_1145_3640336 crossref_primary_10_1016_j_infsof_2024_107656 crossref_primary_10_1007_s10664_024_10579_w crossref_primary_10_1145_3583566 crossref_primary_10_1016_j_knosys_2024_112588 crossref_primary_10_1007_s11219_024_09675_3 crossref_primary_10_1145_3716497 |
| Cites_doi | 10.1109/TVCG.2017.2744878 10.20982/tqmp.08.1.p023 10.1016/j.jss.2021.110935 10.1109/TSE.2014.2357438 10.1145/1134285.1134336 10.1145/3447556.3447567 10.1001/jama.2016.17563 10.1016/j.infsof.2014.11.001 10.1145/1463788.1463819 10.1145/3368089.3409720 10.1145/2931037.2931061 10.1177/1094428110375002 10.14778/3342263.3342633 10.1007/s10664-015-9402-8 10.1145/3338906.3338955 10.1109/TPDS.2020.3030548 10.1145/3395363.3397357 10.1145/1958824.1958887 10.1145/3213846.3213866 10.1037/0033-2909.86.2.420 10.1145/3460319.3464843 10.1001/jama.2016.17216 10.14778/3467861.3467869 10.1145/1242524.1242526 10.1145/3102980.3102995 10.1145/3377811.3380395 10.1007/s10664-013-9258-8 10.1109/TDSC.2004.2 10.1145/3368089.3417050 10.1145/3236024.3236030 10.4271/2016-01-0128 10.1145/1330017.1330019 10.1177/001316447303300309 10.1145/3340544 10.1109/TSE.2016.2576454 10.1145/2000791.2000794 10.1145/2345156.2254075 10.1007/s10664-020-09825-8 10.1145/3126908.3126964 |
| ContentType | Journal Article |
| Copyright | 2022 Elsevier B.V. |
| Copyright_xml | – notice: 2022 Elsevier B.V. |
| DBID | AAYXX CITATION |
| DOI | 10.1016/j.infsof.2022.107004 |
| DatabaseName | CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Business |
| EISSN | 1873-6025 |
| ExternalDocumentID | 10_1016_j_infsof_2022_107004 S0950584922001306 |
| GroupedDBID | --K --M -~X .DC .~1 0R~ 1B1 1~. 1~5 29I 4.4 457 4G. 5GY 5VS 7-5 71M 77K 8P~ 9JN AABNK AACTN AAEDT AAEDW AAIAV AAIKJ AAKOC AALRI AAOAW AAQFI AAQXK AAXUO AAYFN AAYOK ABBOA ABFNM ABFRF ABJNI ABMAC ABTAH ABXDB ABYKQ ACDAQ ACGFO ACGFS ACGOD ACNNM ACRLP ACZNC ADBBV ADEZE ADJOM ADMUD AEBSH AEFWE AEKER AENEX AFKWA AFTJW AGHFR AGUBO AGYEJ AHHHB AHZHX AIALX AIEXJ AIKHN AITUG AJBFU AJOXV ALMA_UNASSIGNED_HOLDINGS AMFUW AMRAJ AOUOD ASPBG AVWKF AXJTR AZFZN BKOJK BKOMP BLXMC CS3 DU5 EBS EFJIC EFLBG EJD EO8 EO9 EP2 EP3 FDB FEDTE FGOYB FIRID FNPLU FYGXN G-Q G8K GBLVA GBOLZ HLZ HVGLF HZ~ IHE J1W KOM LG9 M41 MO0 MS~ N9A O-L O9- OAUVE OZT P-8 P-9 P2P PC. PQQKQ Q38 R2- RIG ROL RPZ SBC SDF SDG SDP SES SEW SPC SPCBC SSV SSZ T5K TWZ UHS UNMZH WH7 WUQ XFK ZY4 ~G- AATTM AAXKI AAYWO AAYXX ABDPE ABWVN ACRPL ACVFH ADCNI ADNMO AEIPS AEUPX AFJKZ AFPUW AFXIZ AGCQF AGQPQ AGRNS AIGII AIIUN AKBMS AKRWK AKYEP ANKPU APXCP BNPGV CITATION SSH |
| ID | FETCH-LOGICAL-c306t-e4272025a221e1ffb2b79a9507c93fb7a4c4d6b75754681efea997331dbab1a03 |
| IEDL.DBID | AIKHN |
| ISSN | 0950-5849 |
| IngestDate | Thu Apr 24 23:09:55 EDT 2025 Tue Jul 01 02:22:04 EDT 2025 Fri Feb 23 02:38:18 EST 2024 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Keywords | Deep learning framework Bug detection Bug characteristics Empirical study |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c306t-e4272025a221e1ffb2b79a9507c93fb7a4c4d6b75754681efea997331dbab1a03 |
| ORCID | 0000-0003-4746-8017 |
| ParticipantIDs | crossref_citationtrail_10_1016_j_infsof_2022_107004 crossref_primary_10_1016_j_infsof_2022_107004 elsevier_sciencedirect_doi_10_1016_j_infsof_2022_107004 |
| ProviderPackageCode | CITATION AAYXX |
| PublicationCentury | 2000 |
| PublicationDate | November 2022 2022-11-00 |
| PublicationDateYYYYMMDD | 2022-11-01 |
| PublicationDate_xml | – month: 11 year: 2022 text: November 2022 |
| PublicationDecade | 2020 |
| PublicationTitle | Information and software technology |
| PublicationYear | 2022 |
| Publisher | Elsevier B.V |
| Publisher_xml | – name: Elsevier B.V |
| References | Shrout, Fleiss (b74) 1979; 86 Guo, Chen, Xie, Ma, Hu, Liu, Liu, Zhao, Li (b92) 2019 M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., Tensorflow: A system for large-scale machine learning, in: 12th {USENIX} Symposium on Operating Systems Design and Implementation, {OSDI} 16, 2016, pp. 265–283. G. Li, S.K.S. Hari, M. Sullivan, T. Tsai, K. Pattabiraman, J. Emer, S.W. Keckler, Understanding error propagation in deep learning neural network (DNN) accelerators and applications, in: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2017, pp. 1–12. Collobert, Bengio, Mariéthoz (b43) 2002 Al-Rfou, Alain, Almahairi, Angermueller, Bahdanau, Ballas, Bastien, Bayer, Belikov, Belopolsky (b24) 2016 Tan, Liu, Li, Wang, Zhou, Zhai (b32) 2014; 19 Y. Feng, Q. Shi, X. Gao, J. Wan, C. Fang, Z. Chen, DeepGini: prioritizing massive tests to enhance the robustness of deep neural networks, in: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2020, pp. 177–188. Blei, Ng, Jordan (b64) 2003; 3 Chen, Thomas, Hassan (b66) 2016; 21 Avizienis, Laprie, Randell, Landwehr (b37) 2004; 1 Thung, Lo, Jiang (b87) 2013 Y. Zhang, L. Ren, L. Chen, Y. Xiong, S.C. Cheung, T. Xie, Detecting numerical bugs in neural network architectures, in: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 826–837. Pickerill, Jungen, Ochodek, Maćkowiak, Staron (b59) 2020; 25 J. Anvik, L. Hiew, G.C. Murphy, Who should fix this bug?, in: Proceedings of the 28th International Conference on Software Engineering, 2006, pp. 361–370. P.J. Guo, T. Zimmermann, N. Nagappan, B. Murphy, “ Not my bug!” and other reasons for software bug report reassignments, in: Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, 2011, pp. 395–404. Jin, Song, Shi, Scherpelz, Lu (b31) 2012; 47 Shi, Wang, Chu (b48) 2018 Chen, Li, Li, Lin, Wang, Wang, Xiao, Xu, Zhang, Zhang (b20) 2015 TensorFlow, TensorBoard visualisation kit for TensorFlow. Retrieved from A Github Inc. [Online]. Available Seaman, Shull, Regardie, Elbert, Feldmann, Guo, Godfrey (b30) 2008 Li, Liu, Liu, Sun, You, Yang, Luan, Gan, Yang, Qian (b49) 2020; 32 Zhang, Ma, Niu, Gao, Song (b65) 2012 Dagenais, Ossher, Bellamy, Robillard, De Vries (b89) 2010 Wan, Lo, Xia, Cai (b93) 2017 Sun, Zhou, Li, Hu, Yang, Li (b8) 2017 Seide, Agarwal (b23) 2016 Saha (b79) 2019 Islam, Pan, Nguyen, Rajan (b11) 2020 Hata, Kula, Ishio, Treude (b60) 2021 Wong, Bressler (b2) 2016; 316 Chollet (b26) 2018 Hassan, Holt (b14) 2000 Hallgren (b75) 2012; 8 Zhou, Roy, Abdolrashidi, Wong, Ma, Xu, Liu, Phothilimtha, Wang, Goldie (b46) 2020; 33 G. Antoniol, K. Ayari, M. Di Penta, F. Khomh, Y.G. Guéhéneuc, Is it a bug or an enhancement? A text-based approach to classify change requests, in: Proceedings of the 2008 Conference of the Center for Advanced Studies on Collaborative Research: Meeting of Minds, 2008, pp. 304–318. Buduma, Locascio (b54) 2017 Gulshan, Peng, Coram, Stumpe, Wu, Narayanaswamy, Venugopalan, Widner, Madams, Cuadros (b1) 2016; 316 Di Franco, Guo, Rubio-González (b29) 2017 M.J. Islam, G. Nguyen, R. Pan, H. Rajan, A comprehensive study on deep learning bug characteristics, in: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2019, pp. 510–520. . L. Nguyen, P. Yu, M. Chowdhury, No! Not another deep learning framework, in: Proceedings of the 16th Workshop on Hot Topics in Operating Systems, 2017, pp. 88–93. S. Tokui, K. Oono, S. Hido, J. Clayton, Chainer: a next-generation open source framework for deep learning, in: Proceedings of Workshop on Machine Learning Systems (LearningSys) in the Twenty-Ninth Annual Conference on Neural Information Processing Systems, Vol. 5, NIPS, 2015, pp. 1–6. Steinmacher, Silva, Gerosa, Redmiles (b36) 2015; 59 Wongsuphasawat, Smilkov, Wexler, Wilson, Mane, Fritz, Krishnan, Viégas, Wattenberg (b51) 2017; 24 Tufano, Watson, Bavota, Penta, White, Poshyvanyk (b77) 2019; 28 A. Goffi, A. Gorla, M.D. Ernst, M. Pezzè, Automatic generation of oracles for exceptional behaviors, in: Proceedings of the 25th International Symposium on Software Testing and Analysis, 2016, pp. 213–224. X.H. Phan, C.T. Nguyen, GibbsLDA++: AC/C++ Implementation of Latent Dirichlet Allocation (LDA), Tech. Rep., 2007. Murphy-Hill, Zimmermann, Bird, Nagappan (b70) 2014; 41 Qureshi, Fang (b88) 2011; 14 Jia, Zhong, Wang, Huang, Lu (b13) 2021; 177 Jia, Shelhamer, Donahue, Karayev, Long, Girshick, Guadarrama, Darrell (b18) 2014 Han, Deng, Xia, Wang, Yin (b58) 2019 Pei, Cao, Yang, Jana (b5) 2017 Huval, Wang, Tandon, Kiske, Song, Pazhayampallil, Andriluka, Rajpurkar, Migimatsu, Cheng-Yue (b4) 2015 Herzig, Just, Zeller (b69) 2013 Noda, Nemoto, Hotta, Tanida, Kikuchi (b80) 2020 Koopman, Wagner (b84) 2016; 4 Maetschke, Tennakoon, Vecchiola, Garnavi (b17) 2017 Chetlur, Woolley, Vandermersch, Cohen, Tran, Catanzaro, Shelhamer (b19) 2014 Vasilescu, Yu, Wang, Devanbu, Filkov (b55) 2015 Cotroneo, De Simone, Iannillo, Natella, Rosiello, Bidokhti (b78) 2019 Grosskurth, Godfrey (b15) 2005 Bojarski, Del Testa, Dworakowski, Firner, Flepp, Goyal, Jackel, Monfort, Muller, Zhang (b3) 2016 Thung, Wang, Lo, Jiang (b7) 2012 Chaudhuri, Das, Narasayya (b56) 2007; 32 Xia, Lo, Ding, Al-Kofahi, Nguyen, Wang (b63) 2016; 43 Anvik, Murphy (b61) 2011; 20 N. Humbatova, G. Jahangirova, G. Bavota, V. Riccio, A. Stocco, P. Tonella, Taxonomy of real faults in deep learning systems, in: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, pp. 1110–1121. Weimer, Necula (b83) 2008; 30 Ma, Zhang, Sun, Xue, Li, Juefei-Xu, Xie, Li, Liu, Zhao (b33) 2018 Tian, Pei, Jana, Ray (b34) 2018 Maalej, Nabil (b62) 2015 Cosentino, Izquierdo, Cabot (b57) 2016 X. Zhang, N. Sun, C. Fang, J. Liu, J. Liu, D. Chai, J. Wang, Z. Chen, Predoo: precision testing of deep learning operators, in: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021, pp. 400–412. Chen, Song, Hu (b42) 2021; 22 Mirhoseini, Goldie, Yazgan, Jiang, Songhori, Wang, Lee, Johnson, Pathak, Bae (b47) 2020 pluskid, Mocha: a pluskid repo. Retrieved from A Github Inc. [Online]. Available Y. Gao, Y. Liu, H. Zhang, Z. Li, Y. Zhu, H. Lin, M. Yang, Estimating gpu memory consumption of deep learning models, in: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 1342–1352. Li, Tan, Wang, Lu, Zhou, Zhai (b90) 2006 Nicholson, Gibson (b25) 2017 Wang, Feng, Wang, Jones, Redmiles (b76) 2020; 29 Y. Zhang, Y. Chen, S.C. Cheung, Y. Xiong, L. Zhang, An empirical study on TensorFlow program bugs, in: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2018, pp. 129–140. Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga (b27) 2019; 32 Kunft, Katsifodimos, Schelter, Breß, Rabl, Markl (b40) 2019; 12 Amodei, Olah, Steinhardt, Christiano, Schulman, Mané (b38) 2016 Y. Gao, W. Dou, F. Qin, C. Gao, D. Wang, J. Wei, R. Huang, L. Zhou, Y. Wu, An empirical study on crash recovery bugs in large-scale distributed systems, in: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2018, pp. 539–550. Looks, Herreshoff, Hutchins, Norvig (b50) 2017 Matter, Kuhn, Nierstrasz (b72) 2009 Fleiss, Cohen (b73) 1973; 33 Hapke, Nelson (b41) 2020 Koutsoukos, Nakandala, Karanasos, Saur, Alonso, Interlandi (b52) 2021; 14 Katz, Barrett, Dill, Julian, Kochenderfer (b6) 2017 Ma, Yu, Wu, Wang (b39) 2019; 1 Matter (10.1016/j.infsof.2022.107004_b72) 2009 Noda (10.1016/j.infsof.2022.107004_b80) 2020 Avizienis (10.1016/j.infsof.2022.107004_b37) 2004; 1 Shrout (10.1016/j.infsof.2022.107004_b74) 1979; 86 10.1016/j.infsof.2022.107004_b9 Chetlur (10.1016/j.infsof.2022.107004_b19) 2014 Hata (10.1016/j.infsof.2022.107004_b60) 2021 Nicholson (10.1016/j.infsof.2022.107004_b25) 2017 Mirhoseini (10.1016/j.infsof.2022.107004_b47) 2020 Jia (10.1016/j.infsof.2022.107004_b18) 2014 10.1016/j.infsof.2022.107004_b44 10.1016/j.infsof.2022.107004_b45 Hapke (10.1016/j.infsof.2022.107004_b41) 2020 10.1016/j.infsof.2022.107004_b86 Gulshan (10.1016/j.infsof.2022.107004_b1) 2016; 316 10.1016/j.infsof.2022.107004_b85 Collobert (10.1016/j.infsof.2022.107004_b43) 2002 10.1016/j.infsof.2022.107004_b82 10.1016/j.infsof.2022.107004_b81 Buduma (10.1016/j.infsof.2022.107004_b54) 2017 Tan (10.1016/j.infsof.2022.107004_b32) 2014; 19 Wongsuphasawat (10.1016/j.infsof.2022.107004_b51) 2017; 24 Fleiss (10.1016/j.infsof.2022.107004_b73) 1973; 33 Thung (10.1016/j.infsof.2022.107004_b87) 2013 Bojarski (10.1016/j.infsof.2022.107004_b3) 2016 Jia (10.1016/j.infsof.2022.107004_b13) 2021; 177 Hassan (10.1016/j.infsof.2022.107004_b14) 2000 Blei (10.1016/j.infsof.2022.107004_b64) 2003; 3 Ma (10.1016/j.infsof.2022.107004_b33) 2018 Grosskurth (10.1016/j.infsof.2022.107004_b15) 2005 Pickerill (10.1016/j.infsof.2022.107004_b59) 2020; 25 Guo (10.1016/j.infsof.2022.107004_b92) 2019 Wong (10.1016/j.infsof.2022.107004_b2) 2016; 316 Zhou (10.1016/j.infsof.2022.107004_b46) 2020; 33 Looks (10.1016/j.infsof.2022.107004_b50) 2017 Cotroneo (10.1016/j.infsof.2022.107004_b78) 2019 Kunft (10.1016/j.infsof.2022.107004_b40) 2019; 12 10.1016/j.infsof.2022.107004_b16 Jin (10.1016/j.infsof.2022.107004_b31) 2012; 47 Dagenais (10.1016/j.infsof.2022.107004_b89) 2010 10.1016/j.infsof.2022.107004_b12 Li (10.1016/j.infsof.2022.107004_b49) 2020; 32 10.1016/j.infsof.2022.107004_b53 10.1016/j.infsof.2022.107004_b10 Wang (10.1016/j.infsof.2022.107004_b76) 2020; 29 Tufano (10.1016/j.infsof.2022.107004_b77) 2019; 28 10.1016/j.infsof.2022.107004_b91 Chen (10.1016/j.infsof.2022.107004_b20) 2015 Katz (10.1016/j.infsof.2022.107004_b6) 2017 Sun (10.1016/j.infsof.2022.107004_b8) 2017 Chollet (10.1016/j.infsof.2022.107004_b26) 2018 Vasilescu (10.1016/j.infsof.2022.107004_b55) 2015 Di Franco (10.1016/j.infsof.2022.107004_b29) 2017 Saha (10.1016/j.infsof.2022.107004_b79) 2019 Al-Rfou (10.1016/j.infsof.2022.107004_b24) 2016 Zhang (10.1016/j.infsof.2022.107004_b65) 2012 10.1016/j.infsof.2022.107004_b28 Cosentino (10.1016/j.infsof.2022.107004_b57) 2016 Hallgren (10.1016/j.infsof.2022.107004_b75) 2012; 8 Chen (10.1016/j.infsof.2022.107004_b66) 2016; 21 10.1016/j.infsof.2022.107004_b68 Huval (10.1016/j.infsof.2022.107004_b4) 2015 Tian (10.1016/j.infsof.2022.107004_b34) 2018 Amodei (10.1016/j.infsof.2022.107004_b38) 2016 10.1016/j.infsof.2022.107004_b22 10.1016/j.infsof.2022.107004_b67 10.1016/j.infsof.2022.107004_b21 Seaman (10.1016/j.infsof.2022.107004_b30) 2008 Chaudhuri (10.1016/j.infsof.2022.107004_b56) 2007; 32 Li (10.1016/j.infsof.2022.107004_b90) 2006 Islam (10.1016/j.infsof.2022.107004_b11) 2020 Koopman (10.1016/j.infsof.2022.107004_b84) 2016; 4 Pei (10.1016/j.infsof.2022.107004_b5) 2017 Han (10.1016/j.infsof.2022.107004_b58) 2019 Paszke (10.1016/j.infsof.2022.107004_b27) 2019; 32 Qureshi (10.1016/j.infsof.2022.107004_b88) 2011; 14 Maetschke (10.1016/j.infsof.2022.107004_b17) 2017 Anvik (10.1016/j.infsof.2022.107004_b61) 2011; 20 Ma (10.1016/j.infsof.2022.107004_b39) 2019; 1 Shi (10.1016/j.infsof.2022.107004_b48) 2018 Herzig (10.1016/j.infsof.2022.107004_b69) 2013 Murphy-Hill (10.1016/j.infsof.2022.107004_b70) 2014; 41 Thung (10.1016/j.infsof.2022.107004_b7) 2012 10.1016/j.infsof.2022.107004_b35 Chen (10.1016/j.infsof.2022.107004_b42) 2021; 22 Steinmacher (10.1016/j.infsof.2022.107004_b36) 2015; 59 Koutsoukos (10.1016/j.infsof.2022.107004_b52) 2021; 14 Weimer (10.1016/j.infsof.2022.107004_b83) 2008; 30 Maalej (10.1016/j.infsof.2022.107004_b62) 2015 Xia (10.1016/j.infsof.2022.107004_b63) 2016; 43 10.1016/j.infsof.2022.107004_b71 Wan (10.1016/j.infsof.2022.107004_b93) 2017 Seide (10.1016/j.infsof.2022.107004_b23) 2016 |
| References_xml | – start-page: 149 year: 2008 end-page: 157 ident: b30 article-title: Defect categorization: making use of a decade of widely varying historical data publication-title: Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement – volume: 33 start-page: 13844 year: 2020 end-page: 13855 ident: b46 article-title: Transferable graph optimizers for ml compilers publication-title: Adv. Neural Inf. Process. Syst. – reference: P.J. Guo, T. Zimmermann, N. Nagappan, B. Murphy, “ Not my bug!” and other reasons for software bug report reassignments, in: Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, 2011, pp. 395–404. – volume: 1 start-page: 105 year: 2019 end-page: 115 ident: b39 article-title: PaddlePaddle: An open-source deep learning platform from industrial practice publication-title: Front. Data Domput. – year: 2017 ident: b50 article-title: Deep learning with dynamic computation graphs – start-page: 2135 year: 2016 ident: b23 article-title: CNTK: Microsoft’s open-source deep-learning toolkit publication-title: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining – year: 2020 ident: b47 article-title: Chip placement with deep reinforcement learning – start-page: 1 year: 2017 end-page: 18 ident: b5 article-title: Deepxplore: Automated whitebox testing of deep learning systems publication-title: Proceedings of the 26th Symposium on Operating Systems Principles – year: 2015 ident: b4 article-title: An empirical evaluation of deep learning on highway driving – volume: 1 start-page: 11 year: 2004 end-page: 33 ident: b37 article-title: Basic concepts and taxonomy of dependable and secure computing publication-title: IEEE Trans. Dependable Secure Comput. – start-page: 275 year: 2010 end-page: 284 ident: b89 article-title: Moving into a new software project landscape publication-title: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1 – start-page: 116 year: 2015 end-page: 125 ident: b62 article-title: Bug report, feature request, or simply praise? on automatically classifying app reviews publication-title: 2015 IEEE 23rd International Requirements Engineering Conference – reference: Y. Gao, W. Dou, F. Qin, C. Gao, D. Wang, J. Wei, R. Huang, L. Zhou, Y. Wu, An empirical study on crash recovery bugs in large-scale distributed systems, in: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2018, pp. 539–550. – volume: 316 start-page: 2402 year: 2016 end-page: 2410 ident: b1 article-title: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs publication-title: JAMA – reference: G. Li, S.K.S. Hari, M. Sullivan, T. Tsai, K. Pattabiraman, J. Emer, S.W. Keckler, Understanding error propagation in deep learning neural network (DNN) accelerators and applications, in: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2017, pp. 1–12. – start-page: 773 year: 2021 end-page: 784 ident: b60 article-title: Same file, different changes: the potential of meta-maintenance on GitHub publication-title: 2021 IEEE/ACM 43rd International Conference on Software Engineering – volume: 29 start-page: 1 year: 2020 end-page: 35 ident: b76 article-title: Unveiling elite developers’ activities in open source projects publication-title: ACM Trans. Softw. Eng. Methodol. – reference: G. Antoniol, K. Ayari, M. Di Penta, F. Khomh, Y.G. Guéhéneuc, Is it a bug or an enhancement? A text-based approach to classify change requests, in: Proceedings of the 2008 Conference of the Center for Advanced Studies on Collaborative Research: Meeting of Minds, 2008, pp. 304–318. – year: 2015 ident: b20 article-title: Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems – volume: 22 start-page: 35 year: 2021 end-page: 50 ident: b42 article-title: Techniques for automated machine learning publication-title: ACM SIGKDD Explor. Newsl. – volume: 43 start-page: 272 year: 2016 end-page: 297 ident: b63 article-title: Improving automated bug triaging with specialized topic model publication-title: IEEE Trans. Softw. Eng. – reference: A. Goffi, A. Gorla, M.D. Ernst, M. Pezzè, Automatic generation of oracles for exceptional behaviors, in: Proceedings of the 25th International Symposium on Software Testing and Analysis, 2016, pp. 213–224. – year: 2002 ident: b43 article-title: Torch: A Modular Machine Learning Software Library – start-page: 1135 year: 2020 end-page: 1146 ident: b11 article-title: Repairing deep neural networks: Fix patterns and challenges publication-title: 2020 IEEE/ACM 42nd International Conference on Software Engineering – start-page: 334 year: 2019 end-page: 345 ident: b78 article-title: Analyzing the context of bug-fixing changes in the openstack cloud computing platform publication-title: 2019 IEEE 30th International Symposium on Software Reliability Engineering – start-page: 661 year: 2005 end-page: 664 ident: b15 article-title: A reference architecture for web browsers publication-title: 21st IEEE International Conference on Software Maintenance – reference: . pluskid, Mocha: a pluskid repo. Retrieved from A Github Inc. [Online]. Available: – volume: 32 start-page: 9 year: 2007 end-page: es ident: b56 article-title: Optimized stratified sampling for approximate query processing publication-title: ACM Trans. Database Syst. – year: 2016 ident: b3 article-title: End to end learning for self-driving cars – start-page: 21 year: 2019 end-page: 26 ident: b58 article-title: Characterization and prediction of popular projects on github publication-title: 2019 IEEE 43rd Annual Computer Software and Applications Conference, Vol. 1 – start-page: 150 year: 2000 end-page: 159 ident: b14 article-title: A reference architecture for web servers publication-title: Proceedings Seventh Working Conference on Reverse Engineering – start-page: 810 year: 2019 end-page: 822 ident: b92 article-title: An empirical study towards characterizing deep learning development and deployment across different frameworks and platforms publication-title: 2019 34th IEEE/ACM International Conference on Automated Software Engineering – start-page: 392 year: 2013 end-page: 401 ident: b69 article-title: It’s not a bug, it’s a feature: how misclassification impacts bug prediction publication-title: 2013 35th International Conference on Software Engineering – reference: N. Humbatova, G. Jahangirova, G. Bavota, V. Riccio, A. Stocco, P. Tonella, Taxonomy of real faults in deep learning systems, in: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, pp. 1110–1121. – start-page: 100 year: 2018 end-page: 111 ident: b33 article-title: Deepmutation: Mutation testing of deep learning systems publication-title: 2018 IEEE 29th International Symposium on Software Reliability Engineering – volume: 25 start-page: 2897 year: 2020 end-page: 2929 ident: b59 article-title: PHANTOM: Curating GitHub for engineered software projects using time-series clustering publication-title: Empir. Softw. Eng. – year: 2016 ident: b38 article-title: Concrete problems in AI safety – start-page: 509 year: 2017 end-page: 519 ident: b29 article-title: A comprehensive study of real-world numerical bug characteristics publication-title: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering – volume: 20 start-page: 1 year: 2011 end-page: 35 ident: b61 article-title: Reducing the effort of bug report triage: Recommenders for development-oriented decisions publication-title: ACM Trans. Softw. Eng. Methodol. – reference: M.J. Islam, G. Nguyen, R. Pan, H. Rajan, A comprehensive study on deep learning bug characteristics, in: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2019, pp. 510–520. – volume: 47 start-page: 77 year: 2012 end-page: 88 ident: b31 article-title: Understanding and detecting real-world performance bugs publication-title: ACM SIGPLAN Not. – reference: Y. Feng, Q. Shi, X. Gao, J. Wan, C. Fang, Z. Chen, DeepGini: prioritizing massive tests to enhance the robustness of deep neural networks, in: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2020, pp. 177–188. – start-page: 675 year: 2014 end-page: 678 ident: b18 article-title: Caffe: Convolutional architecture for fast feature embedding publication-title: Proceedings of the 22nd ACM International Conference on Multimedia – volume: 21 start-page: 1843 year: 2016 end-page: 1919 ident: b66 article-title: A survey on the use of topic models when mining software repositories publication-title: Empir. Softw. Eng. – start-page: 348 year: 2017 end-page: 357 ident: b8 article-title: An empirical study on real bugs for machine learning programs publication-title: 2017 24th Asia-Pacific Software Engineering Conference – volume: 30 start-page: 1 year: 2008 end-page: 51 ident: b83 article-title: Exceptional situations and program reliability publication-title: ACM Trans. Program. Lang. Syst. – year: 2014 ident: b19 article-title: cudnn: Efficient primitives for deep learning – reference: S. Tokui, K. Oono, S. Hido, J. Clayton, Chainer: a next-generation open source framework for deep learning, in: Proceedings of Workshop on Machine Learning Systems (LearningSys) in the Twenty-Ninth Annual Conference on Neural Information Processing Systems, Vol. 5, NIPS, 2015, pp. 1–6. – year: 2017 ident: b25 article-title: Deeplearning4j: Open-source, Distributed Deep Learning for the JVM – volume: 3 start-page: 993 year: 2003 end-page: 1022 ident: b64 article-title: Latent dirichlet allocation publication-title: J. Mach. Learn. Res. – start-page: 805 year: 2015 end-page: 816 ident: b55 article-title: Quality and productivity outcomes relating to continuous integration in GitHub publication-title: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering – year: 2017 ident: b54 article-title: Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms – volume: 28 start-page: 1 year: 2019 end-page: 29 ident: b77 article-title: An empirical study on learning bug-fixing patches in the wild via neural machine translation publication-title: ACM Trans. Softw. Eng. Methodol. – reference: Y. Zhang, Y. Chen, S.C. Cheung, Y. Xiong, L. Zhang, An empirical study on TensorFlow program bugs, in: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2018, pp. 129–140. – volume: 8 start-page: 23 year: 2012 ident: b75 article-title: Computing inter-rater reliability for observational data: an overview and tutorial publication-title: Tutor. Quant. Methods Psychol. – start-page: 612 year: 2020 end-page: 616 ident: b80 article-title: Experience report: how effective is automated program repair for industrial software? publication-title: 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering – volume: 316 start-page: 2366 year: 2016 end-page: 2367 ident: b2 article-title: Artificial intelligence with deep learning technology looks into diabetic retinopathy screening publication-title: JAMA – reference: . TensorFlow, TensorBoard visualisation kit for TensorFlow. Retrieved from A Github Inc. [Online]. Available: – reference: X.H. Phan, C.T. Nguyen, GibbsLDA++: AC/C++ Implementation of Latent Dirichlet Allocation (LDA), Tech. Rep., 2007. – start-page: 97 year: 2017 end-page: 117 ident: b6 article-title: Reluplex: An efficient SMT solver for verifying deep neural networks publication-title: International Conference on Computer Aided Verification – volume: 12 start-page: 1553 year: 2019 end-page: 1567 ident: b40 article-title: An intermediate representation for optimizing machine learning pipelines publication-title: Proc. VLDB Endow. – year: 2018 ident: b26 article-title: Keras: The Python Deep Learning Library – volume: 41 start-page: 65 year: 2014 end-page: 81 ident: b70 article-title: The design space of bug fixes and how developers navigate it publication-title: IEEE Trans. Softw. Eng. – volume: 33 start-page: 613 year: 1973 end-page: 619 ident: b73 article-title: The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability publication-title: Educ. Psychol. Meas. – reference: L. Nguyen, P. Yu, M. Chowdhury, No! Not another deep learning framework, in: Proceedings of the 16th Workshop on Hot Topics in Operating Systems, 2017, pp. 88–93. – start-page: 413 year: 2017 end-page: 424 ident: b93 article-title: Bug characteristics in blockchain systems: a large-scale empirical study publication-title: 2017 IEEE/ACM 14th International Conference on Mining Software Repositories – volume: 4 start-page: 15 year: 2016 end-page: 24 ident: b84 article-title: Challenges in autonomous vehicle testing and validation publication-title: SAE Int. J. Transp. Saf. – year: 2020 ident: b41 article-title: Building Machine Learning Pipelines – volume: 19 start-page: 1665 year: 2014 end-page: 1705 ident: b32 article-title: Bug characteristics in open source software publication-title: Empir. Softw. Eng. – reference: X. Zhang, N. Sun, C. Fang, J. Liu, J. Liu, D. Chai, J. Wang, Z. Chen, Predoo: precision testing of deep learning operators, in: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021, pp. 400–412. – volume: 14 start-page: 1797 year: 2021 end-page: 1804 ident: b52 article-title: Tensors: An abstraction for general data processing publication-title: Proc. VLDB Endow. – start-page: 1309 year: 2012 end-page: 1314 ident: b65 article-title: Multi-document summarization of product reviews publication-title: 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery – start-page: 949 year: 2018 end-page: 957 ident: b48 article-title: Performance modeling and evaluation of distributed deep learning frameworks on gpus publication-title: 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress – start-page: 303 year: 2018 end-page: 314 ident: b34 article-title: Deeptest: Automated testing of deep-neural-network-driven autonomous cars publication-title: Proceedings of the 40th International Conference on Software Engineering – reference: Y. Gao, Y. Liu, H. Zhang, Z. Li, Y. Zhu, H. Lin, M. Yang, Estimating gpu memory consumption of deep learning models, in: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 1342–1352. – volume: 24 start-page: 1 year: 2017 end-page: 12 ident: b51 article-title: Visualizing dataflow graphs of deep learning models in tensorflow publication-title: IEEE Trans. Vis. Comput. Graphics – volume: 177 year: 2021 ident: b13 article-title: The symptoms, causes, and repairs of bugs inside a deep learning library publication-title: J. Syst. Softw. – volume: 32 start-page: 708 year: 2020 end-page: 727 ident: b49 article-title: The deep learning compiler: A comprehensive survey publication-title: IEEE Trans. Parallel Distrib. Syst. – year: 2016 ident: b24 article-title: Theano: A Python framework for fast computation of mathematical expressions – volume: 59 start-page: 67 year: 2015 end-page: 85 ident: b36 article-title: A systematic literature review on the barriers faced by newcomers to open source software projects publication-title: Inf. Softw. Technol. – start-page: 131 year: 2009 end-page: 140 ident: b72 article-title: Assigning bug reports using a vocabulary-based expertise model of developers publication-title: 2009 6th IEEE International Working Conference on Mining Software Repositories – volume: 14 start-page: 208 year: 2011 end-page: 238 ident: b88 article-title: Socialization in open source software projects: A growth mixture modeling approach publication-title: Organ. Res. Methods – volume: 32 start-page: 8026 year: 2019 end-page: 8037 ident: b27 article-title: Pytorch: An imperative style, high-performance deep learning library publication-title: Adv. Neural Inf. Process. Syst. – reference: . – start-page: 271 year: 2012 end-page: 280 ident: b7 article-title: An empirical study of bugs in machine learning systems publication-title: 2012 IEEE 23rd International Symposium on Software Reliability Engineering – year: 2017 ident: b17 article-title: nuts-flow/ml: data pre-processing for deep learning – start-page: 137 year: 2016 end-page: 141 ident: b57 article-title: Findings from GitHub: methods, datasets and limitations publication-title: 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories – reference: J. Anvik, L. Hiew, G.C. Murphy, Who should fix this bug?, in: Proceedings of the 28th International Conference on Software Engineering, 2006, pp. 361–370. – reference: Y. Zhang, L. Ren, L. Chen, Y. Xiong, S.C. Cheung, T. Xie, Detecting numerical bugs in neural network architectures, in: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 826–837. – start-page: 25 year: 2006 end-page: 33 ident: b90 article-title: Have things changed now?: an empirical study of bug characteristics in modern open source software publication-title: Proceedings of the 1st Workshop on Architectural and System Support for Improving Software Dependability – volume: 86 start-page: 420 year: 1979 ident: b74 article-title: Intraclass correlations: uses in assessing rater reliability publication-title: Psychol. Bull. – reference: M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., Tensorflow: A system for large-scale machine learning, in: 12th {USENIX} Symposium on Operating Systems Design and Implementation, {OSDI} 16, 2016, pp. 265–283. – start-page: 92 year: 2013 end-page: 101 ident: b87 article-title: Automatic recovery of root causes from bug-fixing changes publication-title: 2013 20th Working Conference on Reverse Engineering – start-page: 13 year: 2019 end-page: 24 ident: b79 article-title: Harnessing evolution for multi-hunk program repair publication-title: 2019 IEEE/ACM 41st International Conference on Software Engineering – start-page: 303 year: 2018 ident: 10.1016/j.infsof.2022.107004_b34 article-title: Deeptest: Automated testing of deep-neural-network-driven autonomous cars – volume: 24 start-page: 1 issue: 1 year: 2017 ident: 10.1016/j.infsof.2022.107004_b51 article-title: Visualizing dataflow graphs of deep learning models in tensorflow publication-title: IEEE Trans. Vis. Comput. Graphics doi: 10.1109/TVCG.2017.2744878 – ident: 10.1016/j.infsof.2022.107004_b22 – volume: 32 start-page: 8026 year: 2019 ident: 10.1016/j.infsof.2022.107004_b27 article-title: Pytorch: An imperative style, high-performance deep learning library publication-title: Adv. Neural Inf. Process. Syst. – start-page: 334 year: 2019 ident: 10.1016/j.infsof.2022.107004_b78 article-title: Analyzing the context of bug-fixing changes in the openstack cloud computing platform – start-page: 773 year: 2021 ident: 10.1016/j.infsof.2022.107004_b60 article-title: Same file, different changes: the potential of meta-maintenance on GitHub – ident: 10.1016/j.infsof.2022.107004_b45 – start-page: 392 year: 2013 ident: 10.1016/j.infsof.2022.107004_b69 article-title: It’s not a bug, it’s a feature: how misclassification impacts bug prediction – year: 2014 ident: 10.1016/j.infsof.2022.107004_b19 – start-page: 137 year: 2016 ident: 10.1016/j.infsof.2022.107004_b57 article-title: Findings from GitHub: methods, datasets and limitations – year: 2017 ident: 10.1016/j.infsof.2022.107004_b25 – volume: 8 start-page: 23 issue: 1 year: 2012 ident: 10.1016/j.infsof.2022.107004_b75 article-title: Computing inter-rater reliability for observational data: an overview and tutorial publication-title: Tutor. Quant. Methods Psychol. doi: 10.20982/tqmp.08.1.p023 – volume: 177 year: 2021 ident: 10.1016/j.infsof.2022.107004_b13 article-title: The symptoms, causes, and repairs of bugs inside a deep learning library publication-title: J. Syst. Softw. doi: 10.1016/j.jss.2021.110935 – volume: 1 start-page: 105 issue: 1 year: 2019 ident: 10.1016/j.infsof.2022.107004_b39 article-title: PaddlePaddle: An open-source deep learning platform from industrial practice publication-title: Front. Data Domput. – volume: 41 start-page: 65 issue: 1 year: 2014 ident: 10.1016/j.infsof.2022.107004_b70 article-title: The design space of bug fixes and how developers navigate it publication-title: IEEE Trans. Softw. Eng. doi: 10.1109/TSE.2014.2357438 – ident: 10.1016/j.infsof.2022.107004_b71 doi: 10.1145/1134285.1134336 – start-page: 150 year: 2000 ident: 10.1016/j.infsof.2022.107004_b14 article-title: A reference architecture for web servers – volume: 22 start-page: 35 issue: 2 year: 2021 ident: 10.1016/j.infsof.2022.107004_b42 article-title: Techniques for automated machine learning publication-title: ACM SIGKDD Explor. Newsl. doi: 10.1145/3447556.3447567 – year: 2016 ident: 10.1016/j.infsof.2022.107004_b3 – start-page: 131 year: 2009 ident: 10.1016/j.infsof.2022.107004_b72 article-title: Assigning bug reports using a vocabulary-based expertise model of developers – volume: 316 start-page: 2366 issue: 22 year: 2016 ident: 10.1016/j.infsof.2022.107004_b2 article-title: Artificial intelligence with deep learning technology looks into diabetic retinopathy screening publication-title: JAMA doi: 10.1001/jama.2016.17563 – start-page: 13 year: 2019 ident: 10.1016/j.infsof.2022.107004_b79 article-title: Harnessing evolution for multi-hunk program repair – volume: 59 start-page: 67 year: 2015 ident: 10.1016/j.infsof.2022.107004_b36 article-title: A systematic literature review on the barriers faced by newcomers to open source software projects publication-title: Inf. Softw. Technol. doi: 10.1016/j.infsof.2014.11.001 – start-page: 100 year: 2018 ident: 10.1016/j.infsof.2022.107004_b33 article-title: Deepmutation: Mutation testing of deep learning systems – start-page: 116 year: 2015 ident: 10.1016/j.infsof.2022.107004_b62 article-title: Bug report, feature request, or simply praise? on automatically classifying app reviews – ident: 10.1016/j.infsof.2022.107004_b68 doi: 10.1145/1463788.1463819 – start-page: 25 year: 2006 ident: 10.1016/j.infsof.2022.107004_b90 article-title: Have things changed now?: an empirical study of bug characteristics in modern open source software – start-page: 2135 year: 2016 ident: 10.1016/j.infsof.2022.107004_b23 article-title: CNTK: Microsoft’s open-source deep-learning toolkit – start-page: 149 year: 2008 ident: 10.1016/j.infsof.2022.107004_b30 article-title: Defect categorization: making use of a decade of widely varying historical data – ident: 10.1016/j.infsof.2022.107004_b82 doi: 10.1145/3368089.3409720 – start-page: 805 year: 2015 ident: 10.1016/j.infsof.2022.107004_b55 article-title: Quality and productivity outcomes relating to continuous integration in GitHub – ident: 10.1016/j.infsof.2022.107004_b44 – ident: 10.1016/j.infsof.2022.107004_b85 doi: 10.1145/2931037.2931061 – year: 2020 ident: 10.1016/j.infsof.2022.107004_b41 – volume: 14 start-page: 208 issue: 1 year: 2011 ident: 10.1016/j.infsof.2022.107004_b88 article-title: Socialization in open source software projects: A growth mixture modeling approach publication-title: Organ. Res. Methods doi: 10.1177/1094428110375002 – start-page: 97 year: 2017 ident: 10.1016/j.infsof.2022.107004_b6 article-title: Reluplex: An efficient SMT solver for verifying deep neural networks – start-page: 661 year: 2005 ident: 10.1016/j.infsof.2022.107004_b15 article-title: A reference architecture for web browsers – volume: 12 start-page: 1553 issue: 11 year: 2019 ident: 10.1016/j.infsof.2022.107004_b40 article-title: An intermediate representation for optimizing machine learning pipelines publication-title: Proc. VLDB Endow. doi: 10.14778/3342263.3342633 – start-page: 413 year: 2017 ident: 10.1016/j.infsof.2022.107004_b93 article-title: Bug characteristics in blockchain systems: a large-scale empirical study – volume: 21 start-page: 1843 issue: 5 year: 2016 ident: 10.1016/j.infsof.2022.107004_b66 article-title: A survey on the use of topic models when mining software repositories publication-title: Empir. Softw. Eng. doi: 10.1007/s10664-015-9402-8 – ident: 10.1016/j.infsof.2022.107004_b10 doi: 10.1145/3338906.3338955 – volume: 32 start-page: 708 issue: 3 year: 2020 ident: 10.1016/j.infsof.2022.107004_b49 article-title: The deep learning compiler: A comprehensive survey publication-title: IEEE Trans. Parallel Distrib. Syst. doi: 10.1109/TPDS.2020.3030548 – start-page: 271 year: 2012 ident: 10.1016/j.infsof.2022.107004_b7 article-title: An empirical study of bugs in machine learning systems – start-page: 949 year: 2018 ident: 10.1016/j.infsof.2022.107004_b48 article-title: Performance modeling and evaluation of distributed deep learning frameworks on gpus – ident: 10.1016/j.infsof.2022.107004_b35 doi: 10.1145/3395363.3397357 – ident: 10.1016/j.infsof.2022.107004_b86 doi: 10.1145/1958824.1958887 – start-page: 1 year: 2017 ident: 10.1016/j.infsof.2022.107004_b5 article-title: Deepxplore: Automated whitebox testing of deep learning systems – start-page: 509 year: 2017 ident: 10.1016/j.infsof.2022.107004_b29 article-title: A comprehensive study of real-world numerical bug characteristics – start-page: 92 year: 2013 ident: 10.1016/j.infsof.2022.107004_b87 article-title: Automatic recovery of root causes from bug-fixing changes – start-page: 675 year: 2014 ident: 10.1016/j.infsof.2022.107004_b18 article-title: Caffe: Convolutional architecture for fast feature embedding – start-page: 1309 year: 2012 ident: 10.1016/j.infsof.2022.107004_b65 article-title: Multi-document summarization of product reviews – start-page: 1135 year: 2020 ident: 10.1016/j.infsof.2022.107004_b11 article-title: Repairing deep neural networks: Fix patterns and challenges – start-page: 348 year: 2017 ident: 10.1016/j.infsof.2022.107004_b8 article-title: An empirical study on real bugs for machine learning programs – ident: 10.1016/j.infsof.2022.107004_b9 doi: 10.1145/3213846.3213866 – start-page: 810 year: 2019 ident: 10.1016/j.infsof.2022.107004_b92 article-title: An empirical study towards characterizing deep learning development and deployment across different frameworks and platforms – volume: 86 start-page: 420 issue: 2 year: 1979 ident: 10.1016/j.infsof.2022.107004_b74 article-title: Intraclass correlations: uses in assessing rater reliability publication-title: Psychol. Bull. doi: 10.1037/0033-2909.86.2.420 – year: 2015 ident: 10.1016/j.infsof.2022.107004_b20 – volume: 33 start-page: 13844 year: 2020 ident: 10.1016/j.infsof.2022.107004_b46 article-title: Transferable graph optimizers for ml compilers publication-title: Adv. Neural Inf. Process. Syst. – ident: 10.1016/j.infsof.2022.107004_b81 doi: 10.1145/3460319.3464843 – volume: 316 start-page: 2402 issue: 22 year: 2016 ident: 10.1016/j.infsof.2022.107004_b1 article-title: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs publication-title: JAMA doi: 10.1001/jama.2016.17216 – start-page: 275 year: 2010 ident: 10.1016/j.infsof.2022.107004_b89 article-title: Moving into a new software project landscape – year: 2017 ident: 10.1016/j.infsof.2022.107004_b50 – volume: 14 start-page: 1797 issue: 10 year: 2021 ident: 10.1016/j.infsof.2022.107004_b52 article-title: Tensors: An abstraction for general data processing publication-title: Proc. VLDB Endow. doi: 10.14778/3467861.3467869 – volume: 32 start-page: 9 issue: 2 year: 2007 ident: 10.1016/j.infsof.2022.107004_b56 article-title: Optimized stratified sampling for approximate query processing publication-title: ACM Trans. Database Syst. doi: 10.1145/1242524.1242526 – ident: 10.1016/j.infsof.2022.107004_b16 doi: 10.1145/3102980.3102995 – ident: 10.1016/j.infsof.2022.107004_b12 doi: 10.1145/3377811.3380395 – volume: 19 start-page: 1665 issue: 6 year: 2014 ident: 10.1016/j.infsof.2022.107004_b32 article-title: Bug characteristics in open source software publication-title: Empir. Softw. Eng. doi: 10.1007/s10664-013-9258-8 – volume: 1 start-page: 11 issue: 1 year: 2004 ident: 10.1016/j.infsof.2022.107004_b37 article-title: Basic concepts and taxonomy of dependable and secure computing publication-title: IEEE Trans. Dependable Secure Comput. doi: 10.1109/TDSC.2004.2 – year: 2020 ident: 10.1016/j.infsof.2022.107004_b47 – start-page: 21 year: 2019 ident: 10.1016/j.infsof.2022.107004_b58 article-title: Characterization and prediction of popular projects on github – year: 2016 ident: 10.1016/j.infsof.2022.107004_b38 – ident: 10.1016/j.infsof.2022.107004_b67 – ident: 10.1016/j.infsof.2022.107004_b21 – ident: 10.1016/j.infsof.2022.107004_b53 doi: 10.1145/3368089.3417050 – ident: 10.1016/j.infsof.2022.107004_b91 doi: 10.1145/3236024.3236030 – volume: 4 start-page: 15 issue: 1 year: 2016 ident: 10.1016/j.infsof.2022.107004_b84 article-title: Challenges in autonomous vehicle testing and validation publication-title: SAE Int. J. Transp. Saf. doi: 10.4271/2016-01-0128 – volume: 30 start-page: 1 issue: 2 year: 2008 ident: 10.1016/j.infsof.2022.107004_b83 article-title: Exceptional situations and program reliability publication-title: ACM Trans. Program. Lang. Syst. doi: 10.1145/1330017.1330019 – start-page: 612 year: 2020 ident: 10.1016/j.infsof.2022.107004_b80 article-title: Experience report: how effective is automated program repair for industrial software? – volume: 33 start-page: 613 issue: 3 year: 1973 ident: 10.1016/j.infsof.2022.107004_b73 article-title: The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability publication-title: Educ. Psychol. Meas. doi: 10.1177/001316447303300309 – year: 2002 ident: 10.1016/j.infsof.2022.107004_b43 – year: 2017 ident: 10.1016/j.infsof.2022.107004_b54 – year: 2017 ident: 10.1016/j.infsof.2022.107004_b17 – volume: 28 start-page: 1 issue: 4 year: 2019 ident: 10.1016/j.infsof.2022.107004_b77 article-title: An empirical study on learning bug-fixing patches in the wild via neural machine translation publication-title: ACM Trans. Softw. Eng. Methodol. doi: 10.1145/3340544 – volume: 43 start-page: 272 issue: 3 year: 2016 ident: 10.1016/j.infsof.2022.107004_b63 article-title: Improving automated bug triaging with specialized topic model publication-title: IEEE Trans. Softw. Eng. doi: 10.1109/TSE.2016.2576454 – year: 2016 ident: 10.1016/j.infsof.2022.107004_b24 – year: 2018 ident: 10.1016/j.infsof.2022.107004_b26 – year: 2015 ident: 10.1016/j.infsof.2022.107004_b4 – volume: 3 start-page: 993 year: 2003 ident: 10.1016/j.infsof.2022.107004_b64 article-title: Latent dirichlet allocation publication-title: J. Mach. Learn. Res. – volume: 20 start-page: 1 issue: 3 year: 2011 ident: 10.1016/j.infsof.2022.107004_b61 article-title: Reducing the effort of bug report triage: Recommenders for development-oriented decisions publication-title: ACM Trans. Softw. Eng. Methodol. doi: 10.1145/2000791.2000794 – volume: 47 start-page: 77 issue: 6 year: 2012 ident: 10.1016/j.infsof.2022.107004_b31 article-title: Understanding and detecting real-world performance bugs publication-title: ACM SIGPLAN Not. doi: 10.1145/2345156.2254075 – volume: 29 start-page: 1 issue: 3 year: 2020 ident: 10.1016/j.infsof.2022.107004_b76 article-title: Unveiling elite developers’ activities in open source projects publication-title: ACM Trans. Softw. Eng. Methodol. – volume: 25 start-page: 2897 year: 2020 ident: 10.1016/j.infsof.2022.107004_b59 article-title: PHANTOM: Curating GitHub for engineered software projects using time-series clustering publication-title: Empir. Softw. Eng. doi: 10.1007/s10664-020-09825-8 – ident: 10.1016/j.infsof.2022.107004_b28 doi: 10.1145/3126908.3126964 |
| SSID | ssj0017030 |
| Score | 2.494803 |
| Snippet | Deep Learning (DL) frameworks enable developers to build DNN models without learning the underlying algorithms and models. While some of these DL-based... |
| SourceID | crossref elsevier |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 107004 |
| SubjectTerms | Bug characteristics Bug detection Deep learning framework Empirical study |
| Title | A comprehensive empirical study on bug characteristics of deep learning frameworks |
| URI | https://dx.doi.org/10.1016/j.infsof.2022.107004 |
| Volume | 151 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1LS8NAEB76APEiPrE-yh68rk120yR7LMVSFXtQC72FfaVWNAl9XP3t7mYTsSAKHrPshOzsZGZgvvkG4EpTKjyWKsyp1DhgxPxSQnk4pYqoWPkyLHkKHibheBrczfqzBgzrXhgLq6x8v_PppbeuVnqVNnvFYtF7MsmBZ8InI6Qsv4VNaBPKwn4L2oPb-_Hkq5hgjdpR7nnYCtQddCXMy9zjKrdcnoSYJUv2_nOE-hZ1RvuwV6WLaOC-6AAaOjuEnRqtfgSPA2RB4Uv94oDoSL8Xi5L1A5XEsSjPkNjMkdzmZUZ5ipTWBaqGRsxRWoO0VscwHd08D8e4GpOApTn1GuvA1lJJnxPiaz9NBRER4-awkWQ0FREPZKBCEZnELAhjX6eaM2ZHNSrBhc89egKtLM_0KSCfmhcEivk8Mqme2RcLFgdc8jiSmnPRAVqrJpEVh7gdZfGW1GCx18QpNLEKTZxCO4C_pArHofHH_qjWerJlC4lx879Knv1b8hx27ZPrMryA1nq50Zcm3ViLLjSvP_xuZVSfR17VpA |
| linkProvider | Elsevier |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1LT8MwDI7GkIAL4inGMweuYW2Stc1xmpgGbDvAJu0W5dUxBG21x5XfTtLHxCQEEtfUrhrHtS3l82cAbg0h0mOxRoIogyjD9peS2kMx0VhH2ldBzlMwGAa9MX2ctCY10Kl6YRyssoz9RUzPo3W50iyt2cxms-aLLQ48mz4Zxvn1W7AFtmmLhA7Xd_e5xnn4zqULwj0POfGqfy4HedlTXKSOyRNju-So3n_OT99yTvcA7JfFImwX33MIaiY5AjsVVv0YPLehg4TPzWsBQ4fmI5vlnB8wp42FaQLlagrVJiszTGOojclgOTJiCuMKorU4AePu_ajTQ-WQBKTsnpfIUHeTilsCY9_4cSyxDJmwmw0VI7EMBVVUBzK0ZRkNIt_ERjDmBjVqKaQvPHIK6kmamDMAfWJfQDXzRWgLPSsXSRZRoUQUKiOEbABSmYarkkHcDbJ45xVU7I0XBuXOoLwwaAOgtVZWMGj8IR9WVucbnsBtkP9V8_zfmjdgtzca9Hn_Yfh0Afbck6Lf8BLUl_OVubKFx1Je5471BROK1m8 |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+comprehensive+empirical+study+on+bug+characteristics+of+deep+learning+frameworks&rft.jtitle=Information+and+software+technology&rft.au=Yang%2C+Yilin&rft.au=He%2C+Tianxing&rft.au=Xia%2C+Zhilong&rft.au=Feng%2C+Yang&rft.date=2022-11-01&rft.issn=0950-5849&rft.volume=151&rft.spage=107004&rft_id=info:doi/10.1016%2Fj.infsof.2022.107004&rft.externalDBID=n%2Fa&rft.externalDocID=10_1016_j_infsof_2022_107004 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0950-5849&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0950-5849&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0950-5849&client=summon |