Can Offline Testing of Deep Neural Networks Replace Their Online Testing? A Case Study of Automated Driving Systems
We distinguish two general modes of testing for Deep Neural Networks (DNNs): Offline testing where DNNs are tested as individual units based on test datasets obtained without involving the DNNs under test, and online testing where DNNs are embedded into a specific application environment and tested...
Saved in:
| Published in | Empirical software engineering : an international journal Vol. 26; no. 5; p. 90 |
|---|---|
| Main Authors | , , , |
| Format | Journal Article |
| Language | English |
| Published |
New York
Springer US
01.09.2021
|
| Subjects | |
| Online Access | Get full text |
Cover
Loading…
| Abstract | We distinguish two general modes of testing for Deep Neural Networks (DNNs): Offline testing where DNNs are tested as individual units based on test datasets obtained without involving the DNNs under test, and online testing where DNNs are embedded into a specific application environment and tested in a closed-loop mode in interaction with the application environment. Typically, DNNs are subjected to both types of testing during their development life cycle where offline testing is applied immediately after DNN training and online testing follows after offline testing and once a DNN is deployed within a specific application environment. In this paper, we study the relationship between offline and online testing. Our goal is to determine
how offline testing and online testing differ or complement one another
and
if offline testing results can be used to help reduce the cost of online testing?
Though these questions are generally relevant to all autonomous systems, we study them in the context of automated driving systems where, as study subjects, we use DNNs automating end-to-end controls of steering functions of self-driving vehicles. Our results show that offline testing is less effective than online testing as many safety violations identified by online testing could not be identified by offline testing, while large prediction errors generated by offline testing always led to severe safety violations detectable by online testing. Further, we cannot exploit offline testing results to reduce the cost of online testing in practice since we are not able to identify specific situations where offline testing could be as accurate as online testing in identifying safety requirement violations. |
|---|---|
| AbstractList | We distinguish two general modes of testing for Deep Neural Networks (DNNs): Offline testing where DNNs are tested as individual units based on test datasets obtained without involving the DNNs under test, and online testing where DNNs are embedded into a specific application environment and tested in a closed-loop mode in interaction with the application environment. Typically, DNNs are subjected to both types of testing during their development life cycle where offline testing is applied immediately after DNN training and online testing follows after offline testing and once a DNN is deployed within a specific application environment. In this paper, we study the relationship between offline and online testing. Our goal is to determine
how offline testing and online testing differ or complement one another
and
if offline testing results can be used to help reduce the cost of online testing?
Though these questions are generally relevant to all autonomous systems, we study them in the context of automated driving systems where, as study subjects, we use DNNs automating end-to-end controls of steering functions of self-driving vehicles. Our results show that offline testing is less effective than online testing as many safety violations identified by online testing could not be identified by offline testing, while large prediction errors generated by offline testing always led to severe safety violations detectable by online testing. Further, we cannot exploit offline testing results to reduce the cost of online testing in practice since we are not able to identify specific situations where offline testing could be as accurate as online testing in identifying safety requirement violations. We distinguish two general modes of testing for Deep Neural Networks (DNNs): Offline testing where DNNs are tested as individual units based on test datasets obtained without involving the DNNs under test, and online testing where DNNs are embedded into a specific application environment and tested in a closed-loop mode in interaction with the application environment. Typically, DNNs are subjected to both types of testing during their development life cycle where offline testing is applied immediately after DNN training and online testing follows after offline testing and once a DNN is deployed within a specific application environment. In this paper, we study the relationship between offline and online testing. Our goal is to determine how offline testing and online testing differ or complement one another and if offline testing results can be used to help reduce the cost of online testing? Though these questions are generally relevant to all autonomous systems, we study them in the context of automated driving systems where, as study subjects, we use DNNs automating end-to-end controls of steering functions of self-driving vehicles. Our results show that offline testing is less effective than online testing as many safety violations identified by online testing could not be identified by offline testing, while large prediction errors generated by offline testing always led to severe safety violations detectable by online testing. Further, we cannot exploit offline testing results to reduce the cost of online testing in practice since we are not able to identify specific situations where offline testing could be as accurate as online testing in identifying safety requirement violations.We distinguish two general modes of testing for Deep Neural Networks (DNNs): Offline testing where DNNs are tested as individual units based on test datasets obtained without involving the DNNs under test, and online testing where DNNs are embedded into a specific application environment and tested in a closed-loop mode in interaction with the application environment. Typically, DNNs are subjected to both types of testing during their development life cycle where offline testing is applied immediately after DNN training and online testing follows after offline testing and once a DNN is deployed within a specific application environment. In this paper, we study the relationship between offline and online testing. Our goal is to determine how offline testing and online testing differ or complement one another and if offline testing results can be used to help reduce the cost of online testing? Though these questions are generally relevant to all autonomous systems, we study them in the context of automated driving systems where, as study subjects, we use DNNs automating end-to-end controls of steering functions of self-driving vehicles. Our results show that offline testing is less effective than online testing as many safety violations identified by online testing could not be identified by offline testing, while large prediction errors generated by offline testing always led to severe safety violations detectable by online testing. Further, we cannot exploit offline testing results to reduce the cost of online testing in practice since we are not able to identify specific situations where offline testing could be as accurate as online testing in identifying safety requirement violations. |
| ArticleNumber | 90 |
| Author | Shin, Donghwan Briand, Lionel Haq, Fitash Ul Nejati, Shiva |
| Author_xml | – sequence: 1 givenname: Fitash Ul orcidid: 0000-0003-2253-9085 surname: Haq fullname: Haq, Fitash Ul organization: SnT, University of Luxembourg – sequence: 2 givenname: Donghwan orcidid: 0000-0002-0840-6449 surname: Shin fullname: Shin, Donghwan email: donghwan.shin@uni.lu organization: SnT, University of Luxembourg – sequence: 3 givenname: Shiva orcidid: 0000-0002-0281-8231 surname: Nejati fullname: Nejati, Shiva organization: SnT, University of Luxembourg, University of Ottawa – sequence: 4 givenname: Lionel orcidid: 0000-0002-1393-1010 surname: Briand fullname: Briand, Lionel organization: SnT, University of Luxembourg, University of Ottawa |
| BookMark | eNp9kclOHDEYhK2IKKwvwKmPXBq8u30JioZVQhkpImfL7fk9mPTYE7sbxNtjMiBBDpzKsusrlVy7aCumCAgdEnxMMFYnhWApeYspabHWHW35F7RDhGKtkkRu1TOrl4wKuY12S7nHGGvFxTe0zYTShGm5g65nNjZz74cQobmFMoa4bJJvzgDWzU-Ysh2qjI8p_ynNL1gP1lXfHYTczON76HQfffV2KHDwqnvo98X57eyqvZlfXs9-3LSOKT22FFO7cIRrIE5xLhaWUcwJs0JI37EeFO0dE0S4Kn3vKfGdVworKXqhes_20PdN7nrqV7BwEMda0qxzWNn8ZJIN5uNLDHdmmR6MplwrimvA0WtATn-n2t6sQnEwDDZCmoqhshOcS8VltXYbq8uplAzeuDDaMaSX5DAYgs3LEmazhKlLmH9LGF5R-h_61vBTiG2gUs1xCdncpynH-p2fUc_GvpwO |
| CitedBy_id | crossref_primary_10_1145_3579642 crossref_primary_10_1145_3638246 crossref_primary_10_3390_s24061963 crossref_primary_10_1002_stvr_1840 crossref_primary_10_1145_3664605 crossref_primary_10_1007_s10664_023_10433_5 crossref_primary_10_1007_s11219_022_09613_1 crossref_primary_10_1007_s10664_024_10458_4 crossref_primary_10_1016_j_simpa_2022_100352 crossref_primary_10_1007_s10664_023_10306_x crossref_primary_10_1109_TSE_2023_3327575 crossref_primary_10_1002_smr_2591 crossref_primary_10_1145_3569935 crossref_primary_10_1145_3635709 crossref_primary_10_1145_3643671 crossref_primary_10_1007_s10664_024_10515_y crossref_primary_10_1145_3715105 crossref_primary_10_1109_TSE_2024_3420816 |
| Cites_doi | 10.1145/3238147.3238202 10.1109/ICSE.2019.00108 10.1007/978-3-319-89960-2_22 10.1109/TSE.2014.2372785 10.1007/978-3-319-67361-5_40 10.1145/3180155.3180220 10.1109/IVS.2018.8500421 10.1109/TSE.2019.2962027 10.1007/978-3-030-01267-0_15 10.1109/ITSC45102.2020.9294422 10.1109/QRS.2017.25 10.1016/B978-1-55860-377-6.50023-2 10.1145/3293882.3330566 10.1145/3241979 10.1016/j.tra.2016.09.010 10.1016/j.csda.2007.08.015 10.1109/ICASSP.2013.6639344 10.1016/j.patrec.2010.03.014 10.1109/CVPR.2012.6248110 10.1145/3132747.3132785 10.1145/3238147.3238187 10.1109/ICST46399.2020.00019 10.1177/154193120004402026 10.1109/CVPR.2012.6248074 |
| ContentType | Journal Article |
| Copyright | The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2021. corrected publication 2022 The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2021, corrected publication 2022. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2021, corrected publication 2022 |
| Copyright_xml | – notice: The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2021. corrected publication 2022 – notice: The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2021, corrected publication 2022. – notice: The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2021, corrected publication 2022 |
| DBID | C6C AAYXX CITATION 7X8 5PM |
| DOI | 10.1007/s10664-021-09982-4 |
| DatabaseName | Springer Nature OA Free Journals CrossRef MEDLINE - Academic PubMed Central (Full Participant titles) |
| DatabaseTitle | CrossRef MEDLINE - Academic |
| DatabaseTitleList | CrossRef MEDLINE - Academic |
| Database_xml | – sequence: 1 dbid: C6C name: Springer Nature OA Free Journals url: http://www.springeropen.com/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISSN | 1573-7616 |
| ExternalDocumentID | PMC9249720 10_1007_s10664_021_09982_4 |
| GrantInformation_xml | – fundername: Fonds National de la Recherche Luxembourg grantid: BRIDGES2020/IS/14711346/FUNTASY funderid: https://doi.org/10.13039/501100001866 – fundername: Ministry of Education grantid: 2019R1A6A3A03033444 funderid: https://doi.org/10.13039/501100002701 – fundername: IEE S.A. Luxembourg – fundername: H2020 European Research Council grantid: 694277 funderid: https://doi.org/10.13039/100010663 – fundername: Canada Research Chairs funderid: https://doi.org/10.13039/501100001804 – fundername: Natural Sciences and Engineering Research Council of Canada funderid: https://doi.org/10.13039/501100000038 – fundername: ; – fundername: ; grantid: BRIDGES2020/IS/14711346/FUNTASY – fundername: ; grantid: 694277 – fundername: ; grantid: 2019R1A6A3A03033444 |
| GroupedDBID | -4Z -59 -5G -BR -EM -Y2 -~C .86 .DC .VR 06D 0R~ 0VY 199 1N0 1SB 2.D 203 28- 29G 2J2 2JN 2JY 2KG 2LR 2P1 2VQ 2~H 30V 4.4 406 408 409 40D 40E 5GY 5QI 5VS 67Z 6NX 78A 8FE 8FG 8TC 8UJ 95- 95. 95~ 96X AABHQ AACDK AAHNG AAIAL AAJBT AAJKR AANZL AAOBN AARHV AARTL AASML AATNV AATVU AAUYE AAWCG AAYIU AAYOK AAYQN AAYTO AAYZH ABAKF ABBBX ABBXA ABDZT ABECU ABFTD ABFTV ABHLI ABHQN ABJCF ABJNI ABJOX ABKCH ABKTR ABMNI ABMQK ABNWP ABQBU ABQSL ABSXP ABTEG ABTHY ABTKH ABTMW ABULA ABWNU ABXPI ACAOD ACBXY ACDTI ACGFS ACHSB ACHXU ACIWK ACKNC ACMDZ ACMLO ACOKC ACOMO ACPIV ACSNA ACZOJ ADHHG ADHIR ADIMF ADINQ ADKNI ADKPE ADRFC ADTPH ADURQ ADYFF ADZKW AEBTG AEFIE AEFQL AEGAL AEGNC AEJHL AEJRE AEKMD AEMSY AENEX AEOHA AEPYU AESKC AETLH AEVLU AEXYK AFBBN AFEXP AFGCZ AFKRA AFLOW AFQWF AFWTZ AFZKB AGAYW AGDGC AGGDS AGJBK AGMZJ AGQEE AGQMX AGRTI AGWIL AGWZB AGYKE AHAVH AHBYD AHKAY AHSBF AHYZX AIAKS AIGIU AIIXL AILAN AITGF AJBLW AJRNO AJZVZ ALMA_UNASSIGNED_HOLDINGS ALWAN AMKLP AMXSW AMYLF AMYQR AOCGG ARAPS ARMRJ ASPBG AVWKF AXYYD AYJHY AZFZN B-. BA0 BBWZM BDATZ BENPR BGLVJ BGNMA BSONS C6C CAG CCPQU COF CS3 CSCUP DDRTE DL5 DNIVK DPUIP DU5 EBLON EBS EIOEI EJD ESBYG FEDTE FERAY FFXSO FIGPU FINBP FNLPD FRRFC FSGXE FWDCC GGCAI GGRSB GJIRD GNWQR GQ6 GQ7 GQ8 GXS H13 HCIFZ HF~ HG5 HG6 HMJXF HQYDN HRMNR HVGLF HZ~ I09 IHE IJ- IKXTQ ITM IWAJR IXC IZIGR IZQ I~X I~Z J-C J0Z JBSCW JCJTX JZLTJ KDC KOV KOW L6V LAK LLZTM M4Y M7S MA- N2Q NB0 NDZJH NPVJJ NQJWS NU0 O9- O93 O9G O9I O9J OAM P19 P62 P9O PF0 PT4 PT5 PTHSS Q2X QOK QOS R4E R89 R9I RHV RNI RNS ROL RPX RSV RZC RZE RZK S0W S16 S1Z S26 S27 S28 S3B SAP SCJ SCLPG SCO SDH SDM SHX SISQX SJYHP SNE SNPRN SNX SOHCF SOJ SPISZ SRMVM SSLCW STPWE SZN T13 T16 TSG TSK TSV TUC U2A UG4 UOJIU UTJUX UZXMN VC2 VFIZW W23 W48 WK8 YLTOR Z45 Z7R Z7S Z7V Z7X Z7Z Z81 Z83 Z86 Z88 Z8M Z8N Z8P Z8R Z8T Z8U Z8W Z92 ZMTXR ~EX AAPKM AAYXX ABBRH ABDBE ABFSG ACSTC ADHKG AEZWR AFDZB AFHIU AFOHR AGQPQ AHPBZ AHWEU AIXLP ATHPR AYFIA CITATION PHGZM PHGZT 7X8 ABRTQ 5PM |
| ID | FETCH-LOGICAL-c379t-202adc149e1c7445da320413a556f83be72bc3515cbc3bbf21f8f770765b57bf3 |
| IEDL.DBID | U2A |
| ISSN | 1382-3256 1573-7616 |
| IngestDate | Thu Aug 21 18:02:21 EDT 2025 Fri Jul 11 16:09:18 EDT 2025 Tue Jul 01 03:32:19 EDT 2025 Thu Apr 24 22:59:58 EDT 2025 Fri Feb 21 02:46:25 EST 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 5 |
| Keywords | Self-driving Cars Deep Learning Testing |
| Language | English |
| License | Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c379t-202adc149e1c7445da320413a556f83be72bc3515cbc3bbf21f8f770765b57bf3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 Communicated by: Miryung Kim |
| ORCID | 0000-0002-0840-6449 0000-0002-0281-8231 0000-0002-1393-1010 0000-0003-2253-9085 |
| OpenAccessLink | https://link.springer.com/10.1007/s10664-021-09982-4 |
| PMID | 35791396 |
| PQID | 2685446746 |
| PQPubID | 23479 |
| ParticipantIDs | pubmedcentral_primary_oai_pubmedcentral_nih_gov_9249720 proquest_miscellaneous_2685446746 crossref_citationtrail_10_1007_s10664_021_09982_4 crossref_primary_10_1007_s10664_021_09982_4 springer_journals_10_1007_s10664_021_09982_4 |
| ProviderPackageCode | CITATION AAYXX |
| PublicationCentury | 2000 |
| PublicationDate | 2021-09-01 |
| PublicationDateYYYYMMDD | 2021-09-01 |
| PublicationDate_xml | – month: 09 year: 2021 text: 2021-09-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York |
| PublicationSubtitle | An International Journal |
| PublicationTitle | Empirical software engineering : an international journal |
| PublicationTitleAbbrev | Empir Software Eng |
| PublicationYear | 2021 |
| Publisher | Springer US |
| Publisher_xml | – name: Springer US |
| References | KalraNPaddockSMDriving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?Trans Res Part A Pol Pract20169418219310.1016/j.tra.2016.09.010https://doi.org/10.1016/j.tra.2016.09.010. http://www.sciencedirect.com/science/article/pii/S0965856416302129 Kim J, Feldt R, Yoo S (2019) Guiding deep learning system testing using surprise adequacy. In: Proceedings of the 41st international conference on software engineering, IEEE Press, Piscataway, NJ, USA, ICSE ’19, pp 1039–1049 https://doi.org/10.1109/ICSE.2019.00108 Ma L, Juefei-Xu F, Zhang F, Sun J, Xue M, Li B, Chen C, Su T, Li L, Liu Y, Zhao J, Wang Y (2018) Deepgauge: Multi-granularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ACM, New York, NY, USA, ASE, 2018, pp 120–131. https://doi.org/10.1145/3238147.3238202 Dreossi T, Ghosh S, Sangiovanni-Vincentelli A, Seshia SA (2017) Systematic testing of convolutional neural networks for autonomous driving. arXiv:1708.03309 Udacity (2016a) Udacity self-driving car challenge 2: Using deep learning to predict steering angles. https://github.com/udacity/self-driving-car/tree/master/challenges/challenge-2, Accessed: 2019-10-11 Gambi A, Mueller M, Fraser G (2019) Automatically testing self-driving cars with search-based procedural content generation. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ACM, New York, NY, USA, ISSTA, 2019 pp 318–328. https://doi.org/10.1145/3293882.3330566 McGehee DV, Mazzae EN, Baldwin GS (2000) Driver reaction time in crash avoidance research: Validation of a driving simulator study on a test track. In: Proceedings of the human factors and ergonomics society annual meeting 44(20):3–320–3–323 https://doi.org/10.1177/154193120004402026 Zhou H, Li W, Zhu Y, Zhang Y, Yu B, Zhang L, Liu C (2018) Deepbillboard: Systematic physical-world testing of autonomous driving systems. arXiv:1812.10812 Sotiropoulos T, Waeselynck H, Guiochet J, Ingrand F (2017) Can robot navigation bugs be found in simulation? an exploratory study. In: 2017 IEEE International conference on software quality, reliability and security (QRS), pp 150–159 https://doi.org/10.1109/QRS.2017.25 Codevilla F, Lopez AM, Koltun V, Dosovitskiy A (2018) On offline evaluation of vision-based driving models. In: The european conference on computer vision (ECCV) Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: Automated whitebox testing of deep learning systems. In: Proceedings of the 26th symposium on operating systems principles, ACM, New York, NY, USA, SOSP ’17, pp 1–18 https://doi.org/10.1145/3132747.3132785 ZhouZQSunLMetamorphic testing of driverless carsCommun ACM2019623616710.1145/3241979https://doi.org/10.1145/3241979 Zhang M, Zhang Y, Zhang L, Liu C, Khurshid S (2018) Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, acm, New York, NY, USA, ASE 2018, pp 132–142. https://doi.org/10.1145/3238147.3238187 Tian Y, Pei K, Jana S, Ray B (2018) Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th international conference on software engineering, ACM, New York, NY, USA, ICSE ’18, pp 303–314. https://doi.org/10.1145/3180155.3180220 Haq FU, Shin D, Nejati S, Briand L (2020b) Supporting materials (temporal link for the double-blind review). http://tiny.cc/Experiment-data, Accessed: 2020-07-26 GenuerRPoggiJMTuleau-MalotCVariable selection using random forestsPattern Recognit Lett201031142225223610.1016/j.patrec.2010.03.014https://doi.org/10.1016/j.patrec.2010.03.014. http://www.sciencedirect.com/science/article/pii/S0167865510000954 Group OM (2014) Object constraint language specification. https://www.omg.org/spec/OCL/, Accessed: 2019-10-11 Deng L, Hinton G, Kingsbury B (2013) New types of deep neural network learning for speech recognition and related applications: an overview. In: 2013 IEEE International conference on acoustics, speech and signal processing, pp 8599–8603 https://doi.org/10.1109/ICASSP.2013.6639344 Autumn T (2016) Autumn model. https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/autumn, Accessed: 2019-10-11 Ciresan DC, Meier U, Schmidhuber J (2012) Multi-column deep neural networks for image classification. arXiv:1202.2745 TASS International - Siemens Group (2019) Prescan: Simulation of adas and active safety. https://tass.plm.automation.siemens.com, Accessed: 2019-10-11 Dosovitskiy A, Ros G, Codevilla F, Lopez A, Koltun V (2017) CARLA: An open urban driving simulator. In: Proceedings of the 1st annual conference on robot learning, pp 1–16 Haq FU, Shin D, Nejati S, Briand L (2020a) Comparing offline and online testing of deep neural networks: An autonomous car case study. In: 2020 IEEE International conference on software testing, verification and validation, p to appear BarrETHarmanMMcMinnPShahbazMYooSThe oracle problem in software testing: A surveyIEEE Trans Softw Eng201541550752510.1109/TSE.2014.2372785https://doi.org/10.1109/TSE.2014.2372785 Cohen WW (1995) Fast effective rule induction. In: Prieditis A, Russell S (eds) Machine Learning Proceedings 1995, Morgan Kaufmann, San Francisco (CA), pp 115–123 https://doi.org/10.1016/B978-1-55860-377-6.50023-2. http://www.sciencedirect.com/science/article/pii/B9781558603776500232 Pineau J (2019) Icse 2019 keynote: Building reproducible, reusable, and robust machine learning software. https://2019.icse-conferences.org/details/icse-2019-Plenary-Sessions/20/Building-Reproducible-Reusable-and-Robust-Machine-Learning-Software, Accessed: 2019-10-11 Rong G, Shin BH, Tabatabaee H, Lu Q, Lemke S, Mozeikǒ M, Boise E, Uhm G, Gerow M, Mehta S, Agafonov E, Kim TH, Sterner E, Ushiroda K, Reyes M, Zelenkovsky D, Kim S (2020) Lgsvl simulator: A high fidelity simulator for autonomous driving. In: 2020 IEEE 23rd International conference on intelligent transportation systems (ITSC), pp 1–6 https://doi.org/10.1109/ITSC45102.2020.9294422 Shah S, Dey D, Lovett C, Kapoor A (2018) Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In: Hutter M, Siegwart R (eds) Field and service robotics, springer international publishing, Cham, pp 621–635 Zhang JM, Harman M, Ma L, Liu Y (2020) Machine learning testing: Survey, landscapes and horizons. IEEE Trans Softw Eng 1–1 Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? the kitti vision benchmark suite ESI Group (2019) Esi pro-sivic - 3d simulations of environments and sensors. https://www.esi-group.com/software-solutions/virtual-environment/virtual-systems-controls/esi-pro-sivictm-3d-simulations-environments-and-sensors, Accessed: 2019-10-11 Wicker M, Huang X, Kwiatkowska M (2018) Feature-guided black-box safety testing of deep neural networks. In: Beyer D, Huisman M (eds) Tools and Algorithms for the Construction and Analysis of Systems. Springer International Publishing, Cham, pp 408–426 Chauffeur T (2016) Chauffeur model. https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/chauffeur, Accessed: 2019-10-11 Pomerleau DA (1989) Alvinn: An autonomous land vehicle in a neural network. In: Advances in neural information processing systems, pp 305–313 Komanda T (2016) Komanda model. https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/komanda, Accessed: 2020-04-14 Tuncali CE, Fainekos G, Ito H, Kapinski J (2018) Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In: 2018 IEEE intelligent vehicles symposium, IV pp 1555–1562. https://doi.org/10.1109/IVS.2018.8500421 Archer KJ, Kimes RV (2008) Empirical characterization of random forest variable importance measures, vol 52. https://doi.org/10.1016/j.csda.2007.08.015. http://www.sciencedirect.com/science/article/pii/S0167947307003076 Majumdar R, Mathur A, Pirron M, Stegner L, Zufferey D (2019) Paracosm: A language and tool for testing autonomous driving systems. arXiv:1902.01084 Udacity (2016b) Udacity self-driving challenge 2, ch2-001 (testing) and ch2-002 (training). https://github.com/udacity/self-driving-car/tree/master/datasets/CH2, Accessed: 2019-10-11 Sutskever I, Vinyals O, Le Q V (2014) Sequence to sequence learning with neural networks. In: Ghahramani Z, Welling M, Cortes C, Lawrence N D, Weinberger K Q (eds) Advances in Neural Information Processing Systems. 27 Curran Associates Inc. pp 3104–3112 9982_CR27 ET Barr (9982_CR3) 2015; 41 9982_CR26 9982_CR29 9982_CR28 9982_CR21 9982_CR20 9982_CR2 9982_CR23 9982_CR22 9982_CR25 9982_CR1 R Genuer (9982_CR14) 2010; 31 9982_CR24 9982_CR6 9982_CR7 9982_CR4 9982_CR5 9982_CR8 9982_CR9 9982_CR16 9982_CR38 9982_CR15 9982_CR37 9982_CR17 9982_CR39 N Kalra (9982_CR18) 2016; 94 ZQ Zhou (9982_CR40) 2019; 62 9982_CR19 9982_CR30 9982_CR10 9982_CR32 9982_CR31 9982_CR12 9982_CR34 9982_CR11 9982_CR33 9982_CR36 9982_CR13 9982_CR35 |
| References_xml | – reference: Dosovitskiy A, Ros G, Codevilla F, Lopez A, Koltun V (2017) CARLA: An open urban driving simulator. In: Proceedings of the 1st annual conference on robot learning, pp 1–16 – reference: Shah S, Dey D, Lovett C, Kapoor A (2018) Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In: Hutter M, Siegwart R (eds) Field and service robotics, springer international publishing, Cham, pp 621–635 – reference: Zhang M, Zhang Y, Zhang L, Liu C, Khurshid S (2018) Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, acm, New York, NY, USA, ASE 2018, pp 132–142. https://doi.org/10.1145/3238147.3238187 – reference: Udacity (2016a) Udacity self-driving car challenge 2: Using deep learning to predict steering angles. https://github.com/udacity/self-driving-car/tree/master/challenges/challenge-2, Accessed: 2019-10-11 – reference: TASS International - Siemens Group (2019) Prescan: Simulation of adas and active safety. https://tass.plm.automation.siemens.com, Accessed: 2019-10-11 – reference: Pineau J (2019) Icse 2019 keynote: Building reproducible, reusable, and robust machine learning software. https://2019.icse-conferences.org/details/icse-2019-Plenary-Sessions/20/Building-Reproducible-Reusable-and-Robust-Machine-Learning-Software, Accessed: 2019-10-11 – reference: Rong G, Shin BH, Tabatabaee H, Lu Q, Lemke S, Mozeikǒ M, Boise E, Uhm G, Gerow M, Mehta S, Agafonov E, Kim TH, Sterner E, Ushiroda K, Reyes M, Zelenkovsky D, Kim S (2020) Lgsvl simulator: A high fidelity simulator for autonomous driving. In: 2020 IEEE 23rd International conference on intelligent transportation systems (ITSC), pp 1–6 https://doi.org/10.1109/ITSC45102.2020.9294422 – reference: BarrETHarmanMMcMinnPShahbazMYooSThe oracle problem in software testing: A surveyIEEE Trans Softw Eng201541550752510.1109/TSE.2014.2372785https://doi.org/10.1109/TSE.2014.2372785 – reference: Komanda T (2016) Komanda model. https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/komanda, Accessed: 2020-04-14 – reference: Udacity (2016b) Udacity self-driving challenge 2, ch2-001 (testing) and ch2-002 (training). https://github.com/udacity/self-driving-car/tree/master/datasets/CH2, Accessed: 2019-10-11 – reference: Deng L, Hinton G, Kingsbury B (2013) New types of deep neural network learning for speech recognition and related applications: an overview. In: 2013 IEEE International conference on acoustics, speech and signal processing, pp 8599–8603 https://doi.org/10.1109/ICASSP.2013.6639344 – reference: Group OM (2014) Object constraint language specification. https://www.omg.org/spec/OCL/, Accessed: 2019-10-11 – reference: Sotiropoulos T, Waeselynck H, Guiochet J, Ingrand F (2017) Can robot navigation bugs be found in simulation? an exploratory study. In: 2017 IEEE International conference on software quality, reliability and security (QRS), pp 150–159 https://doi.org/10.1109/QRS.2017.25 – reference: Chauffeur T (2016) Chauffeur model. https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/chauffeur, Accessed: 2019-10-11 – reference: Kim J, Feldt R, Yoo S (2019) Guiding deep learning system testing using surprise adequacy. In: Proceedings of the 41st international conference on software engineering, IEEE Press, Piscataway, NJ, USA, ICSE ’19, pp 1039–1049 https://doi.org/10.1109/ICSE.2019.00108 – reference: Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: Automated whitebox testing of deep learning systems. In: Proceedings of the 26th symposium on operating systems principles, ACM, New York, NY, USA, SOSP ’17, pp 1–18 https://doi.org/10.1145/3132747.3132785 – reference: Zhou H, Li W, Zhu Y, Zhang Y, Yu B, Zhang L, Liu C (2018) Deepbillboard: Systematic physical-world testing of autonomous driving systems. arXiv:1812.10812 – reference: Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? the kitti vision benchmark suite – reference: ESI Group (2019) Esi pro-sivic - 3d simulations of environments and sensors. https://www.esi-group.com/software-solutions/virtual-environment/virtual-systems-controls/esi-pro-sivictm-3d-simulations-environments-and-sensors, Accessed: 2019-10-11 – reference: Haq FU, Shin D, Nejati S, Briand L (2020b) Supporting materials (temporal link for the double-blind review). http://tiny.cc/Experiment-data, Accessed: 2020-07-26 – reference: Archer KJ, Kimes RV (2008) Empirical characterization of random forest variable importance measures, vol 52. https://doi.org/10.1016/j.csda.2007.08.015. http://www.sciencedirect.com/science/article/pii/S0167947307003076 – reference: KalraNPaddockSMDriving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?Trans Res Part A Pol Pract20169418219310.1016/j.tra.2016.09.010https://doi.org/10.1016/j.tra.2016.09.010. http://www.sciencedirect.com/science/article/pii/S0965856416302129 – reference: ZhouZQSunLMetamorphic testing of driverless carsCommun ACM2019623616710.1145/3241979https://doi.org/10.1145/3241979 – reference: Ma L, Juefei-Xu F, Zhang F, Sun J, Xue M, Li B, Chen C, Su T, Li L, Liu Y, Zhao J, Wang Y (2018) Deepgauge: Multi-granularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ACM, New York, NY, USA, ASE, 2018, pp 120–131. https://doi.org/10.1145/3238147.3238202 – reference: Tian Y, Pei K, Jana S, Ray B (2018) Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th international conference on software engineering, ACM, New York, NY, USA, ICSE ’18, pp 303–314. https://doi.org/10.1145/3180155.3180220 – reference: Codevilla F, Lopez AM, Koltun V, Dosovitskiy A (2018) On offline evaluation of vision-based driving models. In: The european conference on computer vision (ECCV) – reference: Haq FU, Shin D, Nejati S, Briand L (2020a) Comparing offline and online testing of deep neural networks: An autonomous car case study. In: 2020 IEEE International conference on software testing, verification and validation, p to appear – reference: GenuerRPoggiJMTuleau-MalotCVariable selection using random forestsPattern Recognit Lett201031142225223610.1016/j.patrec.2010.03.014https://doi.org/10.1016/j.patrec.2010.03.014. http://www.sciencedirect.com/science/article/pii/S0167865510000954 – reference: Cohen WW (1995) Fast effective rule induction. In: Prieditis A, Russell S (eds) Machine Learning Proceedings 1995, Morgan Kaufmann, San Francisco (CA), pp 115–123 https://doi.org/10.1016/B978-1-55860-377-6.50023-2. http://www.sciencedirect.com/science/article/pii/B9781558603776500232 – reference: Zhang JM, Harman M, Ma L, Liu Y (2020) Machine learning testing: Survey, landscapes and horizons. IEEE Trans Softw Eng 1–1 – reference: Pomerleau DA (1989) Alvinn: An autonomous land vehicle in a neural network. In: Advances in neural information processing systems, pp 305–313 – reference: Tuncali CE, Fainekos G, Ito H, Kapinski J (2018) Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In: 2018 IEEE intelligent vehicles symposium, IV pp 1555–1562. https://doi.org/10.1109/IVS.2018.8500421 – reference: Wicker M, Huang X, Kwiatkowska M (2018) Feature-guided black-box safety testing of deep neural networks. In: Beyer D, Huisman M (eds) Tools and Algorithms for the Construction and Analysis of Systems. Springer International Publishing, Cham, pp 408–426 – reference: Majumdar R, Mathur A, Pirron M, Stegner L, Zufferey D (2019) Paracosm: A language and tool for testing autonomous driving systems. arXiv:1902.01084 – reference: Dreossi T, Ghosh S, Sangiovanni-Vincentelli A, Seshia SA (2017) Systematic testing of convolutional neural networks for autonomous driving. arXiv:1708.03309 – reference: Gambi A, Mueller M, Fraser G (2019) Automatically testing self-driving cars with search-based procedural content generation. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ACM, New York, NY, USA, ISSTA, 2019 pp 318–328. https://doi.org/10.1145/3293882.3330566 – reference: McGehee DV, Mazzae EN, Baldwin GS (2000) Driver reaction time in crash avoidance research: Validation of a driving simulator study on a test track. In: Proceedings of the human factors and ergonomics society annual meeting 44(20):3–320–3–323 https://doi.org/10.1177/154193120004402026 – reference: Sutskever I, Vinyals O, Le Q V (2014) Sequence to sequence learning with neural networks. In: Ghahramani Z, Welling M, Cortes C, Lawrence N D, Weinberger K Q (eds) Advances in Neural Information Processing Systems. 27 Curran Associates Inc. pp 3104–3112 – reference: Autumn T (2016) Autumn model. https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/autumn, Accessed: 2019-10-11 – reference: Ciresan DC, Meier U, Schmidhuber J (2012) Multi-column deep neural networks for image classification. arXiv:1202.2745 – ident: 9982_CR21 doi: 10.1145/3238147.3238202 – ident: 9982_CR20 – ident: 9982_CR22 – ident: 9982_CR19 doi: 10.1109/ICSE.2019.00108 – ident: 9982_CR36 doi: 10.1007/978-3-319-89960-2_22 – volume: 41 start-page: 507 issue: 5 year: 2015 ident: 9982_CR3 publication-title: IEEE Trans Softw Eng doi: 10.1109/TSE.2014.2372785 – ident: 9982_CR2 – ident: 9982_CR35 – ident: 9982_CR4 – ident: 9982_CR28 doi: 10.1007/978-3-319-67361-5_40 – ident: 9982_CR39 – ident: 9982_CR34 – ident: 9982_CR32 doi: 10.1145/3180155.3180220 – ident: 9982_CR30 – ident: 9982_CR11 – ident: 9982_CR33 doi: 10.1109/IVS.2018.8500421 – ident: 9982_CR25 – ident: 9982_CR37 doi: 10.1109/TSE.2019.2962027 – ident: 9982_CR6 doi: 10.1007/978-3-030-01267-0_15 – ident: 9982_CR27 doi: 10.1109/ITSC45102.2020.9294422 – ident: 9982_CR29 doi: 10.1109/QRS.2017.25 – ident: 9982_CR7 doi: 10.1016/B978-1-55860-377-6.50023-2 – ident: 9982_CR12 doi: 10.1145/3293882.3330566 – volume: 62 start-page: 61 issue: 3 year: 2019 ident: 9982_CR40 publication-title: Commun ACM doi: 10.1145/3241979 – volume: 94 start-page: 182 year: 2016 ident: 9982_CR18 publication-title: Trans Res Part A Pol Pract doi: 10.1016/j.tra.2016.09.010 – ident: 9982_CR1 doi: 10.1016/j.csda.2007.08.015 – ident: 9982_CR8 doi: 10.1109/ICASSP.2013.6639344 – ident: 9982_CR15 – ident: 9982_CR17 – ident: 9982_CR9 – volume: 31 start-page: 2225 issue: 14 year: 2010 ident: 9982_CR14 publication-title: Pattern Recognit Lett doi: 10.1016/j.patrec.2010.03.014 – ident: 9982_CR5 doi: 10.1109/CVPR.2012.6248110 – ident: 9982_CR31 – ident: 9982_CR24 doi: 10.1145/3132747.3132785 – ident: 9982_CR10 – ident: 9982_CR26 – ident: 9982_CR38 doi: 10.1145/3238147.3238187 – ident: 9982_CR16 doi: 10.1109/ICST46399.2020.00019 – ident: 9982_CR23 doi: 10.1177/154193120004402026 – ident: 9982_CR13 doi: 10.1109/CVPR.2012.6248074 |
| SSID | ssj0009745 |
| Score | 2.4647117 |
| Snippet | We distinguish two general modes of testing for Deep Neural Networks (DNNs): Offline testing where DNNs are tested as individual units based on test datasets... |
| SourceID | pubmedcentral proquest crossref springer |
| SourceType | Open Access Repository Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 90 |
| SubjectTerms | Compilers Computer Science Interpreters Programming Languages Software Engineering/Programming and Operating Systems |
| Subtitle | A Case Study of Automated Driving Systems |
| Title | Can Offline Testing of Deep Neural Networks Replace Their Online Testing? |
| URI | https://link.springer.com/article/10.1007/s10664-021-09982-4 https://www.proquest.com/docview/2685446746 https://pubmed.ncbi.nlm.nih.gov/PMC9249720 |
| Volume | 26 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEB7UXrz4FuujrOBNA8k-k5OU2oeKemlBTyGb7KogabH1_zubR2tFCp5CyO4SZmYz32ZmvgG4EGEkjEyYlwmjPG7xzKpDbbwwMVqkPODUuALnh0c5GPG7Z_FcFYVN62z3OiRZfKl_FLtJyT2XUoCoBnEhX4eGcGd3tOIRbS-odlXRmtiR63kMPXpVKvP3GsvuaIExf2dI_gqTFt6ntwNbFWwk7VLPu7Bm8j3YrlsykGqH7sNtJ8nJk7UOPJKhY9DIX8nYkhtjJsQRceAij2Xm95Qg-HY_0cnQBQtISTpaT7o-gFGvO-wMvKpbgpcyFc3Q3GmSpXjgMUGqOBdZwqiPLioRQtqQaaOoThnClxQvWlsa2NAq5SsptFDaskPYyMe5OQIiLY0E49pPWOQY6XVmMyajLEpZoDPtNyGohRanFZW462jxES9IkJ2gYxR0XAg65k24nM-ZlEQaK0ef17qI0d5dECPJzfhrGlMZCu5apMgmqCUlzZd1jNnLT_L3t4I52x02FcX3v6rVGVdbdrriZY7_N_wENmlpYZ4fnMLG7PPLnCFymekWrIe9fgsa7f7LfRfvOrLTKoz3G72v6C8 |
| linkProvider | Springer Nature |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEF60HvTiW6zPFbxpINlnchKJllbbemmhtyWb7KogabHt_3cnj5YWETzlkOyyzMwy32RmvkHolocRNyKhXsaN9Jh1MasOtfHCxGiesoARAw3Ovb5oD9nLiI8qmhzohVnL30OLmxDMg0ICh2UcGmSbaIu5SBnK92IRLwl2ZTGQGCj1POr8eNUg8_seq05oiSzX6yLXkqOFz2nto90KLOLHUrsHaMPkh2ivHsSAq3t5hDpxkuM3awEy4gHwZuTveGzxkzETDPQbbpN-We89xQ5yw69zPIAUAS6pRutFD8do2HoexG2vmpHgpVRGM2fkJMlSF-aYIJWM8SyhxHeOKeFc2JBqI4lOqQMtqXtobUlgQyulLwXXXGpLT1AjH-fmFGFhScQp035CI-Ch15nNqIiyKKWBzrTfREEtNJVWBOIwx-JLLamPQdDKCVoVglasie4WayYlfcafX9_UulDOyiF1keRmPJ8qIkLOYDCKaCK5oqTFtsCTvfom__wo-LIhxJTEnf--VqeqLur0j8Oc_e_za7TdHvS6qtvpv56jHVJam-cHF6gx-56bS4ddZvqqMNofKizjqg |
| linkToPdf | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA5aQbz4Fuszgjdd3M1z9ySytbQ-qocWegubbKKCbItt_7_JPvpCCp72sJsQZibMNzsz3wBwTcOIapZgL6Wae8TYmFWGUnthoiVVJCBIuwbn1w5r9chTn_bnuvjzavcqJVn0NDiWpmx8N0zN3VzjG2PEc-UFFuFYjEjWwYaNVPJEbcziGe0uz8cUO6I9D1vvXrbN_L3Homua4c3lasmllGnuiZq7YLuEkPCh0PkeWNPZPtipxjPA8rYegHacZPDNGAckYdexaWQfcGBgQ-shdKQcdpNOUQU-ghaIux_qsOsSB7AgIK0W3R-CXvOxG7e8cnKCpzCPxtb0UZIqG_zoQHFCaJpg5Ft3lVDKTIil5kgqbKGMsg8pDQpMaDj3OaOScmnwEahlg0wfA8gMiigm0k9w5NjpZWpSzKI0UjiQqfTrIKiEJlRJK-6mW3yLGSGyE7Swgha5oAWpg5vpmmFBqrHy66tKF8LavktoJJkeTEYCsZASNy6F1QFfUNJ0W8eevfgm-_rMWbRd4MmRPf9tpU5RXt_RisOc_O_zS7D53miKl3bn-RRsocLYPD84A7Xxz0SfW0Azlhe5zf4C_YPr8Q |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Can+Offline+Testing+of+Deep+Neural+Networks+Replace+Their+Online+Testing%3F&rft.jtitle=Empirical+software+engineering+%3A+an+international+journal&rft.au=Haq%2C+Fitash+Ul&rft.au=Shin%2C+Donghwan&rft.au=Nejati%2C+Shiva&rft.au=Briand%2C+Lionel&rft.date=2021-09-01&rft.pub=Springer+US&rft.issn=1382-3256&rft.eissn=1573-7616&rft.volume=26&rft.issue=5&rft_id=info:doi/10.1007%2Fs10664-021-09982-4&rft_id=info%3Apmid%2F35791396&rft.externalDocID=PMC9249720 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1382-3256&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1382-3256&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1382-3256&client=summon |