Mutation Reduction Strategies Considered Harmful
Mutation analysis is a well known yet unfortunately costly method for measuring test suite quality. Researchers have proposed numerous mutation reduction strategies in order to reduce the high cost of mutation analysis, while preserving the representativeness of the original set of mutants. As mutat...
Saved in:
| Published in | IEEE transactions on reliability Vol. 66; no. 3; pp. 854 - 874 |
|---|---|
| Main Authors | , , , , |
| Format | Journal Article |
| Language | English |
| Published |
IEEE
01.09.2017
|
| Subjects | |
| Online Access | Get full text |
Cover
Loading…
| Abstract | Mutation analysis is a well known yet unfortunately costly method for measuring test suite quality. Researchers have proposed numerous mutation reduction strategies in order to reduce the high cost of mutation analysis, while preserving the representativeness of the original set of mutants. As mutation reduction is an area of active research, it is important to understand the limits of possible improvements. We theoretically and empirically investigate the limits of improvement in effectiveness from using mutation reduction strategies compared to random sampling. Using real-world open source programs as subjects, we find an absolute limit in improvement of effectiveness over random sampling- 13.078%. Given our findings with respect to absolute limits, one may ask: How effective are the extant mutation reduction strategies? We evaluate the effectiveness of multiple mutation reduction strategies in comparison to random sampling. We find that none of the mutation reduction strategies evaluated-many forms of operator selection, and stratified sampling (on operators or program elements)-produced an effectiveness advantage larger than 5% in comparison with random sampling. Given the poor performance of mutation selection strategies-they may have a negligible advantage at best, and often perform worse than random sampling- we caution practicing testers against applying mutation reduction strategies without adequate justification. |
|---|---|
| AbstractList | Mutation analysis is a well known yet unfortunately costly method for measuring test suite quality. Researchers have proposed numerous mutation reduction strategies in order to reduce the high cost of mutation analysis, while preserving the representativeness of the original set of mutants. As mutation reduction is an area of active research, it is important to understand the limits of possible improvements. We theoretically and empirically investigate the limits of improvement in effectiveness from using mutation reduction strategies compared to random sampling. Using real-world open source programs as subjects, we find an absolute limit in improvement of effectiveness over random sampling- 13.078%. Given our findings with respect to absolute limits, one may ask: How effective are the extant mutation reduction strategies? We evaluate the effectiveness of multiple mutation reduction strategies in comparison to random sampling. We find that none of the mutation reduction strategies evaluated-many forms of operator selection, and stratified sampling (on operators or program elements)-produced an effectiveness advantage larger than 5% in comparison with random sampling. Given the poor performance of mutation selection strategies-they may have a negligible advantage at best, and often perform worse than random sampling- we caution practicing testers against applying mutation reduction strategies without adequate justification. |
| Author | Gopinath, Rahul Jensen, Carlos Ahmed, Iftekhar Alipour, Mohammad Amin Groce, Alex |
| Author_xml | – sequence: 1 givenname: Rahul surname: Gopinath fullname: Gopinath, Rahul email: rahul@gopinath.org organization: School of Electrical Engineering & Computer Science (EECS), Oregon State University, Corvallis, OR, USA – sequence: 2 givenname: Iftekhar surname: Ahmed fullname: Ahmed, Iftekhar email: ahmedi@oregonstate.edu organization: School of Electrical Engineering & Computer Science (EECS), Oregon State University, Corvallis, OR, USA – sequence: 3 givenname: Mohammad Amin surname: Alipour fullname: Alipour, Mohammad Amin email: alipour@cs.uh.edu organization: Department of Computer Science, University of Houston, Houston, TX, USA – sequence: 4 givenname: Carlos surname: Jensen fullname: Jensen, Carlos email: jensenca@oregonstate.edu organization: School of Electrical Engineering & Computer Science (EECS), Oregon State University, Corvallis, OR, USA – sequence: 5 givenname: Alex surname: Groce fullname: Groce, Alex email: agroce@gmail.com organization: School of Informatics, Computing, and Cyber Systems (SICCS), Northern Arizona University, Flagstaff, AZ, USA |
| BookMark | eNp9j7FOwzAQhi1UJNLCzMCSF0hqO3ZsjygCilSEFMIcOc4ZGaUJspOBtydtKgYGpruT_u_0_Wu06oceELolOCUEq21VphQTkVKBeZ7TCxQRzmVCBCUrFGFMZKI4VVdoHcLnfDKmZITwyzTq0Q19XEI7mdP2Nno9woeDEBdDH1wLHtp4p_3BTt01urS6C3Bznhv0_vhQFbtk__r0XNzvE0NzMiaGc8oaq6kVYME2EoSxYhagGLDRudFNI4CxVrVaWcFkJoxsGbcN5A2TOtsgvvw1fgjBg62NW0xnO9fVBNfH2nVV1sfa9bn2zG3_cF_eHbT__oe4WwgHAL9poRglLM9-AHHPZZA |
| CODEN | IERQAD |
| CitedBy_id | crossref_primary_10_1016_j_infsof_2020_106496 crossref_primary_10_1145_3725528 crossref_primary_10_1109_TSE_2021_3052987 crossref_primary_10_1109_TSE_2019_2903057 crossref_primary_10_3390_sym11091145 crossref_primary_10_1002_stvr_1769 crossref_primary_10_1007_s10664_021_10059_5 crossref_primary_10_1109_TC_2020_2988906 crossref_primary_10_1145_3643756 crossref_primary_10_1016_j_jss_2019_07_100 crossref_primary_10_1007_s10586_023_04074_y crossref_primary_10_1145_3522578 |
| Cites_doi | 10.1109/C-M.1978.218136 10.1007/978-3-319-00945-2_11 10.1016/0164-1212(94)00098-0 10.1145/125489.125473 10.1109/ISSRE.2014.27 10.1145/1062455.1062530 10.1109/ICSE.1993.346062 10.1145/227607.227610 10.1145/2884781.2884787 10.1109/MUTATION.2006.7 10.1002/stvr.4370040104 10.1145/1572272.1572282 10.1002/(SICI)1099-1689(199912)9:4<205::AID-STVR186>3.0.CO;2-X 10.1109/TSE.2006.83 10.1109/CMPSAC.1991.170248 10.1109/SCAM.2008.36 10.1007/978-1-4757-5939-6_7 10.1109/ISSRE.2014.40 10.1145/75309.75324 10.1109/ICST.2012.162 10.1109/ICST.2014.12 10.1016/S0167-6423(03)00022-4 10.1016/j.infsof.2009.04.016 10.1109/ICST.2013.20 10.1145/2635868.2635929 10.1109/ICSTW.2016.45 10.1145/1806799.1806863 10.1145/1145735.1145737 10.1145/2635868.2635920 10.1109/ICST.2014.13 10.1007/978-1-4684-2001-2_9 10.1016/j.jss.2010.07.027 10.1145/285055.285059 10.1109/ICST.2016.22 10.1145/1566445.1566540 10.1007/978-3-642-34691-0_15 10.1145/2786805.2786858 10.1109/MS.2006.91 10.1287/moor.4.3.233 10.1145/2483760.2483774 10.1145/2568225.2568265 10.1002/stvr.226 10.1002/stvr.1486 10.1109/ICSTW.2014.20 10.1002/(SICI)1099-1689(200003)10:1<3::AID-STVR196>3.0.CO;2-P 10.1109/ICST.2011.32 10.1145/2591796.2591884 10.1109/ICST.2017.12 10.1145/567446.567468 10.1016/j.jss.2016.01.007 10.1007/978-3-319-15147-2_33 10.1109/ICSTW.2015.7107454 10.1109/ISSRE.2014.11 10.1109/ICSTW.2009.37 10.1145/1141911.1141916 10.1111/j.1751-5823.2002.tb00178.x 10.1109/ICSM.1999.792604 10.1109/ASE.2013.6693070 10.1017/CBO9780511809163 10.1109/TSE.2005.37 10.1145/1595696.1595750 10.1109/ICSE.2015.24 10.1109/TSE.2010.62 10.1145/229000.226313 |
| ContentType | Journal Article |
| DBID | 97E RIA RIE AAYXX CITATION |
| DOI | 10.1109/TR.2017.2705662 |
| DatabaseName | IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Computer Science |
| EISSN | 1558-1721 |
| EndPage | 874 |
| ExternalDocumentID | 10_1109_TR_2017_2705662 7942146 |
| Genre | orig-research |
| GroupedDBID | -~X .DC 0R~ 29I 4.4 5GY 5VS 6IK 8WZ 97E A6W AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACGFS ACIWK ACNCT AENEX AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD H~9 IAAWW IBMZZ ICLAB IDIHD IFIPE IFJZH IPLJI JAVBF LAI M43 MS~ OCL P2P RIA RIE RNS TN5 VH1 VJK AAYXX CITATION RIG |
| ID | FETCH-LOGICAL-c261t-c5524bfa2f7efefb8e7cf795220e0ca6cabb7e44d9da9f74837c8d45fbe6b48a3 |
| IEDL.DBID | RIE |
| ISSN | 0018-9529 |
| IngestDate | Tue Jul 01 00:49:07 EDT 2025 Thu Apr 24 22:50:53 EDT 2025 Wed Aug 27 02:30:45 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 3 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c261t-c5524bfa2f7efefb8e7cf795220e0ca6cabb7e44d9da9f74837c8d45fbe6b48a3 |
| PageCount | 21 |
| ParticipantIDs | crossref_citationtrail_10_1109_TR_2017_2705662 crossref_primary_10_1109_TR_2017_2705662 ieee_primary_7942146 |
| ProviderPackageCode | CITATION AAYXX |
| PublicationCentury | 2000 |
| PublicationDate | 2017-Sept. 2017-9-00 |
| PublicationDateYYYYMMDD | 2017-09-01 |
| PublicationDate_xml | – month: 09 year: 2017 text: 2017-Sept. |
| PublicationDecade | 2010 |
| PublicationTitle | IEEE transactions on reliability |
| PublicationTitleAbbrev | TR |
| PublicationYear | 2017 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| References | ref57 ref13 ref56 ref15 coles (ref69) 2016 ref58 acree (ref11) 1980 ref14 ref53 lindstrm (ref30) 2016 ref52 ref55 ref10 ref17 ref19 gopinath (ref79) 2015 derezi?ska (ref16) 2013 rothermel (ref64) 1999 ji (ref18) 2009 ref46 ref45 ref48 ref85 coles (ref71) 0 ref44 ref43 ref49 ref8 ref7 ref9 wah (ref35) 2003; 48 strug (ref20) 2012 ref4 ref3 namin (ref54) 2008 ref6 ref5 ref82 ref81 ref40 ref84 ref83 offutt (ref42) 1996 ref80 ref78 ref34 ref36 (ref66) 0 wong (ref47) 1993 ref75 offutt (ref37) 1989; 14 ref77 ref33 ref76 ref32 ref2 ref39 ref38 myers (ref41) 1979 (ref74) 2016 mathur (ref50) 1993 budd (ref12) 1980 mathur (ref31) 2012 lindström (ref59) 2015 lipton (ref1) 1971 ref73 ref72 (ref67) 0 ref68 ref24 ref23 wong (ref51) 1994 ref26 ref25 ref22 hussain (ref63) 2008 ref65 ref21 ref28 ref27 ref29 ammann (ref70) 2015 ref60 ref62 ref61 |
| References_xml | – year: 2012 ident: ref31 publication-title: Foundations of Software Testing – year: 0 ident: ref66 article-title: Software repository – year: 1979 ident: ref41 publication-title: The Art of Software Testing – ident: ref32 doi: 10.1109/C-M.1978.218136 – start-page: 119 year: 2013 ident: ref16 article-title: A quality estimation of mutation clustering in c# programs publication-title: New Results in Dependability and Computer Systems doi: 10.1007/978-3-319-00945-2_11 – ident: ref48 doi: 10.1016/0164-1212(94)00098-0 – ident: ref38 doi: 10.1145/125489.125473 – ident: ref29 doi: 10.1109/ISSRE.2014.27 – ident: ref3 doi: 10.1145/1062455.1062530 – ident: ref9 doi: 10.1109/ICSE.1993.346062 – ident: ref76 doi: 10.1145/227607.227610 – ident: ref28 doi: 10.1145/2884781.2884787 – ident: ref53 doi: 10.1109/MUTATION.2006.7 – ident: ref43 doi: 10.1002/stvr.4370040104 – ident: ref77 doi: 10.1145/1572272.1572282 – ident: ref49 doi: 10.1002/(SICI)1099-1689(199912)9:4<205::AID-STVR186>3.0.CO;2-X – start-page: 351 year: 2008 ident: ref54 article-title: Sufficient mutation operators for measuring test effectiveness publication-title: Proc Int Conf Softw Eng – ident: ref4 doi: 10.1109/TSE.2006.83 – ident: ref46 doi: 10.1109/CMPSAC.1991.170248 – ident: ref62 doi: 10.1109/SCAM.2008.36 – start-page: 449 year: 2015 ident: ref70 article-title: Transforming mutation testing from the technology of the future into the technology of the present publication-title: Proc IEEE Int?l Conf Software Testing Verification and Validation Workshops – ident: ref8 doi: 10.1007/978-1-4757-5939-6_7 – ident: ref40 doi: 10.1109/ISSRE.2014.40 – volume: 14 start-page: 131 year: 1989 ident: ref37 article-title: The coupling effect: Fact or fiction? publication-title: ACM SIGSOFT Softw Eng Notes doi: 10.1145/75309.75324 – ident: ref21 doi: 10.1109/ICST.2012.162 – ident: ref82 doi: 10.1109/ICST.2014.12 – volume: 48 start-page: 119 year: 2003 ident: ref35 article-title: An analysis of the coupling effect I: Single test data publication-title: Sci Comput Program doi: 10.1016/S0167-6423(03)00022-4 – year: 2008 ident: ref63 article-title: Mutation clustering – year: 1971 ident: ref1 article-title: Fault diagnosis of computer programs – start-page: 320 year: 1993 ident: ref50 article-title: Evaluation of the cost of alternate mutation strategies publication-title: Proc Brazilian Symp Software Eng – year: 2015 ident: ref79 article-title: An empirical comparison of mutant selection approaches – ident: ref61 doi: 10.1016/j.infsof.2009.04.016 – year: 1996 ident: ref42 article-title: Subsumption of condition coverage techniques by mutation testing – year: 2016 ident: ref74 article-title: What is the reverse of greedy algorithm for setcover? publication-title: Theoretical Computer Science Stack Exchange – ident: ref56 doi: 10.1109/ICST.2013.20 – year: 0 ident: ref71 article-title: Pit mutation testing: Mutators – start-page: 422 year: 2009 ident: ref18 article-title: A novel method of mutation clustering based on domain analysis publication-title: Proc Int Conf on Software Eng Knowledge Eng – ident: ref5 doi: 10.1145/2635868.2635929 – ident: ref57 doi: 10.1109/ICSTW.2016.45 – year: 1980 ident: ref12 article-title: Mutation analysis of program test data – start-page: 112 year: 2016 ident: ref30 article-title: On strong mutation and subsuming mutants. IEEE workshop on mutation analysis (mutation 2016) publication-title: Proc Workshop on Mutation Analysis – ident: ref13 doi: 10.1145/1806799.1806863 – ident: ref84 doi: 10.1145/1145735.1145737 – ident: ref68 doi: 10.1145/2635868.2635920 – year: 2016 ident: ref69 article-title: Pit mutation testing – ident: ref23 doi: 10.1109/ICST.2014.13 – ident: ref72 doi: 10.1007/978-1-4684-2001-2_9 – year: 1980 ident: ref11 article-title: On mutation – ident: ref39 doi: 10.1016/j.jss.2010.07.027 – ident: ref75 doi: 10.1145/285055.285059 – ident: ref58 doi: 10.1109/ICST.2016.22 – ident: ref55 doi: 10.1145/1566445.1566540 – year: 1993 ident: ref47 article-title: On mutation and data flow – start-page: 200 year: 2012 ident: ref20 article-title: Machine learning approach in mutation testing publication-title: Testing Software and Systems doi: 10.1007/978-3-642-34691-0_15 – ident: ref26 doi: 10.1145/2786805.2786858 – ident: ref7 doi: 10.1109/MS.2006.91 – ident: ref73 doi: 10.1287/moor.4.3.233 – ident: ref24 doi: 10.1145/2483760.2483774 – ident: ref81 doi: 10.1145/2568225.2568265 – ident: ref52 doi: 10.1002/stvr.226 – ident: ref27 doi: 10.1002/stvr.1486 – start-page: 439 year: 1994 ident: ref51 article-title: Constrained mutation in C programs publication-title: Proc Brazilian Symp Software Eng – ident: ref60 doi: 10.1109/ICSTW.2014.20 – ident: ref34 doi: 10.1002/(SICI)1099-1689(200003)10:1<3::AID-STVR196>3.0.CO;2-P – year: 0 ident: ref67 article-title: Apache Maven project – ident: ref25 doi: 10.1109/ICST.2011.32 – ident: ref85 doi: 10.1145/2591796.2591884 – ident: ref36 doi: 10.1109/ICST.2017.12 – ident: ref33 doi: 10.1145/567446.567468 – ident: ref17 doi: 10.1016/j.jss.2016.01.007 – ident: ref15 doi: 10.1007/978-3-319-15147-2_33 – ident: ref19 doi: 10.1109/ICSTW.2015.7107454 – ident: ref6 doi: 10.1109/ISSRE.2014.11 – ident: ref83 doi: 10.1109/ICSTW.2009.37 – ident: ref80 doi: 10.1145/1141911.1141916 – ident: ref78 doi: 10.1111/j.1751-5823.2002.tb00178.x – start-page: 179 year: 1999 ident: ref64 article-title: Test case prioritization: An empirical study publication-title: Proc IEEE Int Conf Softw Maintenance doi: 10.1109/ICSM.1999.792604 – ident: ref14 doi: 10.1109/ASE.2013.6693070 – ident: ref2 doi: 10.1017/CBO9780511809163 – ident: ref22 doi: 10.1109/TSE.2005.37 – ident: ref10 doi: 10.1145/1595696.1595750 – ident: ref65 doi: 10.1109/ICSE.2015.24 – ident: ref45 doi: 10.1109/TSE.2010.62 – ident: ref44 doi: 10.1145/229000.226313 – start-page: 112 year: 2015 ident: ref59 article-title: On redundant mutants and strong mutation publication-title: International Conference on Software Testing Verification and Validation Workshops |
| SSID | ssj0014498 |
| Score | 2.3607347 |
| Snippet | Mutation analysis is a well known yet unfortunately costly method for measuring test suite quality. Researchers have proposed numerous mutation reduction... |
| SourceID | crossref ieee |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 854 |
| SubjectTerms | Computer bugs Computer science Indexes Mutation analysis Redundancy Semantics software testing Upper bound |
| Title | Mutation Reduction Strategies Considered Harmful |
| URI | https://ieeexplore.ieee.org/document/7942146 |
| Volume | 66 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV09T8MwED21nWCg0IIoX8rAwEDSNDixPSJEVSGVoWqlbpHtnBdKi0qy8OuxEycUBBKbFdmSc3fO-XJ37wFcExNVMGNKvjD-xSeCo88jZWFcmVG5kkyWNJ3T52SyIE_LeNmC26YXBhHL4jMM7LDM5WcbVdhfZUNjO5aHug1tE7hVvVpNxoAQ7r665gDHEXcwPqOQD-czW8JFg4gab59E3zzQDqVK6VHGXZjWe6kKSV6CIpeB-vgB0_jfzR7CgbtaeveVLRxBC9c96Na0DZ47xT3Y38Eg7EM4LapsvDezKK7lqIasxXfvi9DTm4jtqy5Wx7AYP84fJr6jUfCVCY9yX8VxRKQWkaaoUUuGForIyCgKMVQiUUJKioRkPBNcUwsxr1hGYi0xkYSJuxPorDdrPAWPEiUwVgkbKUqkjAVLtHnxxDbIZjSWAwhq0abKYYxbqotVWsYaIU_ns9TqInW6GMBNs-Ctgtf4e2rfCrmZ5uR79vvjc9izi6tqsAvo5NsCL831IZdXpd18AhaxwWA |
| linkProvider | IEEE |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3PT4MwFH6Z86AenG4a508OHjwIMmyhPRrjgjp2WFiyG2lLuTg3M-HiX28LBafRxFtDCinvvfL6eO99H8AlUlEFUaZkM-VfbMSotKknNIwrUSoXnPCSpjMa--EUPc3wrAXXTS-MlLIsPpOOHpa5_HQpCv2r7EbZjuah3oBN5ffxoOrWanIGCFHz3VVbGHvUAPkMXHoTT3QRV-B4gfL3vvfNB62RqpQ-ZdiBqF5NVUry4hQ5d8THD6DG_y53D3bN4dK6q6xhH1py0YVOTdxgmX3chZ01FMIeuFFR5eOticZxLUc1aK18t74oPa2QrV6zYn4A0-FDfB_ahkjBFipAym2BsYd4xrwskJnMOJEajEjJyHOlK5gvGOeBRCilKaNZoEHmBUkRzrj0OSLs9hDai-VCHoEVIMEkFj4ZiABxjhnxM_Xivm6RTQPM--DUok2EQRnXZBfzpIw2XJrEk0TrIjG66MNVc8NbBbDx99SeFnIzzcj3-PfLF7AVxtEoGT2On09gWz-oqg07hXa-KuSZOkzk_Ly0oU9ZtcSp |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Mutation+Reduction+Strategies+Considered+Harmful&rft.jtitle=IEEE+transactions+on+reliability&rft.au=Gopinath%2C+Rahul&rft.au=Ahmed%2C+Iftekhar&rft.au=Alipour%2C+Mohammad+Amin&rft.au=Jensen%2C+Carlos&rft.date=2017-09-01&rft.issn=0018-9529&rft.eissn=1558-1721&rft.volume=66&rft.issue=3&rft.spage=854&rft.epage=874&rft_id=info:doi/10.1109%2FTR.2017.2705662&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TR_2017_2705662 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9529&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9529&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9529&client=summon |