Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical Analysis

Context : Both automated and manual software testing are widely applied in practice. While being essential for project success and software quality, they are very resource-intensive, thus motivating the pursuit for optimization. Goal : We aim at understanding to what extent test optimization techniq...

Full description

Saved in:
Bibliographic Details
Published inIEEE transactions on software engineering Vol. 50; no. 8; pp. 2005 - 2020
Main Authors Haas, Roman, Nommer, Raphael, Juergens, Elmar, Apel, Sven
Format Journal Article
LanguageEnglish
Published New York IEEE 01.08.2024
IEEE Computer Society
Subjects
Automation
Codes
Cost benefit analysis
Failure detection
Fault detection
Impact analysis
Industries
manual testing
Manuals
Optimization
Optimization techniques
Software
Software systems
Software testing
test optimization
Testing
Online AccessGet full text

Cover

Abstract Context : Both automated and manual software testing are widely applied in practice. While being essential for project success and software quality, they are very resource-intensive, thus motivating the pursuit for optimization. Goal : We aim at understanding to what extent test optimization techniques for automated testing from the field of test case selection, prioritization, and test suite minimization can be applied to manual testing processes in practice. Method : We have studied the automated and manual testing process of five industrial study subjects from five different domains with different technological backgrounds and assessed the costs and benefits of test optimization techniques in industrial practice. In particular, we have carried out a cost-benefit analysis of two language-agnostic optimization techniques (test impact analysis and Pareto testing a technique we introduce in this paper) on 2,622 real-world failures from our subject's histories. Results : Both techniques maintain most of the fault detection capability while significantly reducing the test runtime. For automated testing, optimized test suites detect, on average, 80% of failures, while saving 66% of execution time, as compared to 81% failure detection rate for manual test suites and an average time saving of 43%. We observe an average speedup of the time to first failure of around 49 compared to a random test ordering. Conclusion : Our results suggest that optimization techniques from automated testing can be transferred to manual testing in industrial practice, resulting in lower test execution time and much lower time-to-feedback, but coming with process-related limitations and requirements for a successful implementation. All study subjects implemented one of our test optimization techniques in their processes, which demonstrates the practical impact of our findings.
AbstractList Context : Both automated and manual software testing are widely applied in practice. While being essential for project success and software quality, they are very resource-intensive, thus motivating the pursuit for optimization. Goal : We aim at understanding to what extent test optimization techniques for automated testing from the field of test case selection, prioritization, and test suite minimization can be applied to manual testing processes in practice. Method : We have studied the automated and manual testing process of five industrial study subjects from five different domains with different technological backgrounds and assessed the costs and benefits of test optimization techniques in industrial practice. In particular, we have carried out a cost–benefit analysis of two language-agnostic optimization techniques (test impact analysis and Pareto testing a technique we introduce in this paper) on 2,622 real-world failures from our subject's histories. Results : Both techniques maintain most of the fault detection capability while significantly reducing the test runtime. For automated testing, optimized test suites detect, on average, 80% of failures, while saving 66% of execution time, as compared to 81% failure detection rate for manual test suites and an average time saving of 43%. We observe an average speedup of the time to first failure of around 49 compared to a random test ordering. Conclusion : Our results suggest that optimization techniques from automated testing can be transferred to manual testing in industrial practice, resulting in lower test execution time and much lower time-to-feedback, but coming with process-related limitations and requirements for a successful implementation. All study subjects implemented one of our test optimization techniques in their processes, which demonstrates the practical impact of our findings.
Author Apel, Sven
Nommer, Raphael
Haas, Roman
Juergens, Elmar
Author_xml – sequence: 1
  givenname: Roman
  orcidid: 0000-0002-3248-3401
  surname: Haas
  fullname: Haas, Roman
  email: haas@cqse.eu
  organization: Saarbrücken Graduate School of Computer Science, Saarbrücken, Germany
– sequence: 2
  givenname: Raphael
  orcidid: 0000-0003-2864-3326
  surname: Nommer
  fullname: Nommer, Raphael
  email: noemmer@cqse.eu
  organization: Saarbrücken Graduate School of Computer Science, Saarbrücken, Germany
– sequence: 3
  givenname: Elmar
  orcidid: 0000-0003-1692-6839
  surname: Juergens
  fullname: Juergens, Elmar
  organization: CQSE GmbH, 81249, Munich, Germany
– sequence: 4
  givenname: Sven
  orcidid: 0000-0003-3687-2233
  surname: Apel
  fullname: Apel, Sven
  organization: Saarland University, Saarland Informatics Campus, 66123, Saarbrücken, Germany
BookMark eNp9kM9LwzAUgINMcJvePXgIeO58SZY28TbGdANlwua5pGkKGV0zk1SZf73dj4N48PQO7_sej2-Aeo1rDEK3BEaEgHxYr2YjCnQ8YmMiiCQXqE8kkwnjFHqoDyBFwrmQV2gQwgYAeJbxPqqXu2i39ltF6xrsKjxpo9uqaEqsmhK_qqZVNV65Kn4pb_DahBiwbfCiKdsQve2Wb17paLV5xBO8av2n2R_VuQ3Reas7YtKoeh9suEaXlaqDuTnPIXp_mq2n8-Rl-byYTl4STSWNiQSqq5RnplSCAWeUG6KETgko0BR0oTVkQmlKlC61AUOhqApZpsAKUWYVG6L7092ddx9t93K-ca3vngg5A0kl4SmIjkpPlPYuBG-qXNt47BC9snVOID-Uzbuy-aFsfi7bifBH3Hm7VX7_n3J3Uqwx5hfOUymFZD-LkYdB
CODEN IESEDJ
CitedBy_id crossref_primary_10_23940_ijpe_25_01_p1_19
Cites_doi 10.1109/IEEESTD.2022.9698145
10.1145/2950290.2950361
10.1109/MS.2017.3571562
10.2197/ipsjjip.30.294
10.1109/ICST57152.2023.00044
10.1145/152388.152391
10.1109/ICSE-SEIP.2017.16
10.1109/32.910860
10.1109/ICST46399.2020.00012
10.1109/ICSE.2015.230
10.1145/3180155.3180210
10.1109/ICSME.2018.00033
10.1145/3460319.3464834
10.1145/3468264.3473922
10.1007/s10664-021-10066-6
10.1109/ISESE.2003.1237978
10.1016/j.infsof.2009.07.001
10.1109/ICMLA.2016.0065
10.1145/3460319.3464810
10.1109/ICST.2018.00038
10.1109/ICSE.1994.296780
10.1145/3395363.3397383
10.1109/ICST57152.2023.00038
10.1145/3213846.3213852
10.1145/2635868.2635910
10.1145/3524481.3527223
10.1109/ICSM.2012.6405252
10.1109/CSMR.2011.44
10.1145/347324.348910
10.1109/ICST.2017.44
10.1145/3213846.3213875
10.1109/TechDebt.2019.00016
10.1109/ICSE.2015.24
10.1109/ICSE48619.2023.00145
10.1109/TestVis57757.2022.00008
10.1145/2593501.2593506
10.1109/ICST.2008.47
10.1109/ISSREW.2019.00033
10.1109/ICSE-SEIP.2019.00018
10.1145/3057269
10.1007/978-3-030-99527-0_12
10.1007/978-3-031-19756-7_17
10.1109/ICST.2011.60
10.1109/TSE.2021.3070549
10.1007/s11219-012-9181-z
10.1109/TSE.2022.3184842
10.1109/ICST.2015.7102602
10.1145/2786805.2786878
10.1002/stvr.1572
10.1145/3241743
10.1109/ICSM.1999.792604
10.1016/j.infsof.2017.08.014
10.1109/ICSE.2019.00054
10.1002/stv.430
10.1145/3524481.3527230
10.1145/2393596.2393634
10.1007/s10664-019-09781-y
10.1109/ICST.2019.00052
10.1145/3106237.3106297
ContentType Journal Article
Copyright Copyright IEEE Computer Society 2024
Copyright_xml – notice: Copyright IEEE Computer Society 2024
DBID 97E
RIA
RIE
AAYXX
CITATION
JQ2
K9.
DOI 10.1109/TSE.2024.3418191
DatabaseName IEEE All-Society Periodicals Package (ASPP) 2005-present
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
ProQuest Computer Science Collection
ProQuest Health & Medical Complete (Alumni)
DatabaseTitle CrossRef
ProQuest Health & Medical Complete (Alumni)
ProQuest Computer Science Collection
DatabaseTitleList ProQuest Health & Medical Complete (Alumni)
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISSN 1939-3520
EndPage 2020
ExternalDocumentID 10_1109_TSE_2024_3418191
10569989
Genre orig-research
GrantInformation_xml – fundername: German Federal Ministry of Education and Research (BMBF)
  grantid: Q-SOFT, 01IS22001A
  funderid: 10.13039/501100002347
– fundername: German Research Foundation (DFG)
  grantid: AP 206/14-1
  funderid: 10.13039/501100001659
GroupedDBID --Z
-DZ
-~X
.4S
.DC
0R~
29I
3EH
4.4
5GY
5VS
6IK
7WY
7X7
85S
88E
88I
8FE
8FG
8FI
8FJ
8FL
8G5
8R4
8R5
97E
9M8
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABFSI
ABJCF
ABPPZ
ABQJQ
ABUWG
ABVLG
ACGFO
ACGOD
ACIWK
ACNCT
ADBBV
AENEX
AETIX
AFKRA
AGQYO
AGSQL
AHBIQ
AI.
AIBXA
AKJIK
AKQYR
ALLEH
ALMA_UNASSIGNED_HOLDINGS
ARAPS
ARCSS
ASUFR
ATWAV
AZQEC
BEFXN
BENPR
BEZIV
BFFAM
BGLVJ
BGNUA
BKEBE
BKOMP
BPEOZ
BPHCQ
BVXVI
CCPQU
CS3
DU5
DWQXO
E.L
EBS
EDO
EJD
FRNLG
FYUFA
GNUQQ
GROUPED_ABI_INFORM_RESEARCH
GUQSH
HCIFZ
HMCUK
HZ~
H~9
I-F
IBMZZ
ICLAB
IEDLZ
IFIPE
IFJZH
IPLJI
ITG
ITH
JAVBF
K60
K6V
K6~
K7-
L6V
LAI
M0C
M1P
M1Q
M2O
M2P
M43
M7S
MS~
O9-
OCL
OHT
P2P
P62
PHGZM
PHGZT
PJZUB
PPXIY
PQBIZ
PQBZA
PQGLB
PQQKQ
PROAC
PSQYO
PTHSS
PUEGO
Q2X
RIA
RIE
RNI
RNS
RXW
RZB
S10
TAE
TN5
TWZ
UHB
UKHRP
UPT
UQL
VH1
WH7
XOL
YYP
YZZ
ZCG
AAYOK
AAYXX
ALIPV
CITATION
RIG
JQ2
K9.
ID FETCH-LOGICAL-c292t-902cf657eda8305325e1a8c610a0c20cbcc078ac21acdce0e20bfb9d603b8d7f3
IEDL.DBID RIE
ISSN 0098-5589
IngestDate Mon Jun 30 10:56:12 EDT 2025
Tue Jul 01 01:53:22 EDT 2025
Thu Apr 24 22:51:12 EDT 2025
Wed Aug 27 02:28:32 EDT 2025
IsPeerReviewed true
IsScholarly true
Issue 8
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
https://doi.org/10.15223/policy-029
https://doi.org/10.15223/policy-037
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c292t-902cf657eda8305325e1a8c610a0c20cbcc078ac21acdce0e20bfb9d603b8d7f3
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ORCID 0000-0002-3248-3401
0000-0003-1692-6839
0000-0003-2864-3326
0000-0003-3687-2233
PQID 3092915608
PQPubID 21418
PageCount 16
ParticipantIDs crossref_citationtrail_10_1109_TSE_2024_3418191
crossref_primary_10_1109_TSE_2024_3418191
proquest_journals_3092915608
ieee_primary_10569989
ProviderPackageCode CITATION
AAYXX
PublicationCentury 2000
PublicationDate 2024-08-01
PublicationDateYYYYMMDD 2024-08-01
PublicationDate_xml – month: 08
  year: 2024
  text: 2024-08-01
  day: 01
PublicationDecade 2020
PublicationPlace New York
PublicationPlace_xml – name: New York
PublicationTitle IEEE transactions on software engineering
PublicationTitleAbbrev TSE
PublicationYear 2024
Publisher IEEE
IEEE Computer Society
Publisher_xml – name: IEEE
– name: IEEE Computer Society
References ref13
ref12
ref56
ref15
ref59
ref14
ref58
ref53
ref52
ref11
ref55
ref54
ref17
ref16
ref19
Marijan (ref57) 2018
ref18
Fowler (ref63)
ref51
ref50
ref46
ref45
ref48
ref47
ref42
ref41
ref44
ref43
ref49
ref8
ref7
ref9
ref4
ref3
ref6
ref5
ref40
ref35
ref34
ref36
ref31
ref30
ref33
Noemmer (ref37) 2019
ref32
ref2
ref1
ref39
(ref10) 2001
ref38
ref24
ref23
ref26
ref25
ref64
ref22
ref21
ref65
(ref60) 2023
ref28
ref27
ref29
Juergens (ref20) 2018
ref62
ref61
References_xml – ident: ref21
  doi: 10.1109/IEEESTD.2022.9698145
– ident: ref15
  doi: 10.1145/2950290.2950361
– ident: ref65
  doi: 10.1109/MS.2017.3571562
– ident: ref52
  doi: 10.2197/ipsjjip.30.294
– ident: ref59
  doi: 10.1109/ICST57152.2023.00044
– ident: ref13
  doi: 10.1145/152388.152391
– ident: ref58
  doi: 10.1109/ICSE-SEIP.2017.16
– ident: ref14
  doi: 10.1109/32.910860
– start-page: 153
  volume-title: Proc. Int. Conf. Softw. Eng.
  year: 2018
  ident: ref57
  article-title: Practical selective regression testing with effective redundancy in interleaved tests
– ident: ref51
  doi: 10.1109/ICST46399.2020.00012
– ident: ref12
  doi: 10.1109/ICSE.2015.230
– start-page: 329
  volume-title: Proc. Int. Conf. Softw. Eng.
  year: 2001
  ident: ref10
  article-title: Incorporating varying test costs and fault severities into test case prioritization
– year: 2019
  ident: ref37
  article-title: Conception and evaluation of test suite minimization techniques for regression testing in practice
– ident: ref39
  doi: 10.1145/3180155.3180210
– ident: ref61
  doi: 10.1109/ICSME.2018.00033
– ident: ref24
  doi: 10.1145/3460319.3464834
– ident: ref6
  doi: 10.1145/3468264.3473922
– ident: ref18
  doi: 10.1007/s10664-021-10066-6
– ident: ref31
  doi: 10.1109/ISESE.2003.1237978
– ident: ref7
  doi: 10.1016/j.infsof.2009.07.001
– ident: ref48
  doi: 10.1109/ICMLA.2016.0065
– ident: ref55
  doi: 10.1145/3460319.3464810
– ident: ref4
  doi: 10.1109/ICST.2018.00038
– ident: ref32
  doi: 10.1109/ICSE.1994.296780
– ident: ref54
  doi: 10.1145/3395363.3397383
– year: 2023
  ident: ref60
  article-title: Microsoft Azure Devops server
– ident: ref23
  doi: 10.1109/ICST57152.2023.00038
– ident: ref49
  doi: 10.1145/3213846.3213852
– ident: ref11
  doi: 10.1145/2635868.2635910
– ident: ref38
  doi: 10.1145/3524481.3527223
– ident: ref30
  doi: 10.1109/ICSM.2012.6405252
– ident: ref42
  doi: 10.1109/CSMR.2011.44
– ident: ref9
  doi: 10.1145/347324.348910
– ident: ref41
  doi: 10.1109/ICST.2017.44
– ident: ref53
  doi: 10.1145/3213846.3213875
– ident: ref35
  doi: 10.1109/TechDebt.2019.00016
– ident: ref64
  doi: 10.1109/ICSE.2015.24
– ident: ref56
  doi: 10.1109/ICSE48619.2023.00145
– ident: ref36
  doi: 10.1109/TestVis57757.2022.00008
– ident: ref43
  doi: 10.1145/2593501.2593506
– ident: ref1
  doi: 10.1109/ICST.2008.47
– ident: ref45
  doi: 10.1109/ISSREW.2019.00033
– ident: ref16
  doi: 10.1109/ICSE-SEIP.2019.00018
– ident: ref27
  doi: 10.1145/3057269
– ident: ref62
  doi: 10.1007/978-3-030-99527-0_12
– ident: ref3
  doi: 10.1007/978-3-031-19756-7_17
– ident: ref26
  doi: 10.1109/ICST.2011.60
– ident: ref50
  doi: 10.1109/TSE.2021.3070549
– ident: ref8
  doi: 10.1007/s11219-012-9181-z
– year: 2018
  ident: ref20
  article-title: Test impact analysis: Detecting errors early despite large, long-running test suites
– ident: ref29
  doi: 10.1109/TSE.2022.3184842
– ident: ref47
  doi: 10.1109/ICST.2015.7102602
– ident: ref40
  doi: 10.1145/2786805.2786878
– ident: ref44
  doi: 10.1002/stvr.1572
– ident: ref19
  doi: 10.1145/3241743
– ident: ref33
  doi: 10.1109/ICSM.1999.792604
– ident: ref22
  doi: 10.1016/j.infsof.2017.08.014
– ident: ref17
  doi: 10.1109/ICSE.2019.00054
– ident: ref5
  doi: 10.1002/stv.430
– ident: ref28
  doi: 10.1145/3524481.3527230
– ident: ref2
  doi: 10.1145/2393596.2393634
– ident: ref34
  doi: 10.1007/s10664-019-09781-y
– ident: ref46
  doi: 10.1109/ICST.2019.00052
– ident: ref63
  article-title: TestPyramid
– ident: ref25
  doi: 10.1145/3106237.3106297
SSID ssj0005775
ssib053395008
Score 2.4463367
Snippet Context : Both automated and manual software testing are widely applied in practice. While being essential for project success and software quality, they are...
SourceID proquest
crossref
ieee
SourceType Aggregation Database
Enrichment Source
Index Database
Publisher
StartPage 2005
SubjectTerms Automation
Codes
Cost benefit analysis
Failure detection
Fault detection
Impact analysis
Industries
manual testing
Manuals
Optimization
Optimization techniques
Software
Software systems
Software testing
test optimization
Testing
Title Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical Analysis
URI https://ieeexplore.ieee.org/document/10569989
https://www.proquest.com/docview/3092915608
Volume 50
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3NS8MwFA9uJy_Oj4nTKTl48dAuS78Sb0M2huAUtsFuJU1TEGcrW6voX-9Lmzo_ULyUQpMS-CWvv9f33u8hdC78SMezqEU4XFzmupaQVFgOE34AbM6VVZbvxB_P3euFtzDF6mUtjFKqTD5Ttr4tY_lxJgv9q6ynu8SDe8AbqAGeW1WstcnnCAKvFsj0PMbrmCThvdl0CJ4gdW0w2dpB-fINKpuq_LDE5edl1EKTemFVVsmDXeSRLd--aTb-e-W7aMcQTTyodsYe2lLpPmrVTRywOdMHaHkLRuPRVGPiLMGDIs-AxqoYizTGN0JrluIpWOsXsVJ4Bmtf4_sUb3p-4DtTaXWJB3harJ7Vazl1o0CCa-mTNpqPhrOrsWVaMFiScppbnFCZ-F6gYsEc3UTCU33BJHAuQSQlMpISOAag2xcyloooSqIk4rFPnIjFQeIcomaapeoI4aQPzwMvUtQJXI8RITkTVIF7GSQ6ltdBvRqUUBp9ct0mYxmWfgrhIcAYahhDA2MHXXzMeKq0Of4Y29aofBpXAdJB3Rr40JzedegQII26xJwd_zLtBG3rt1eZgF3UzFeFOgV2kkdn5a58B3KT4Es
linkProvider IEEE
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3PT9swFH5i5QCXMRho3brhA5cd0rpOnNjcKtSqQFsmtUjcIsdxJDSWTCXZBH_9nhNn3YZAu0SRYiuWPuf5e3k_PoATFSY2nsU8KvESiCDwlGbK84UKI2RzgW6yfBfh9Dq4uOE3rli9roUxxtTJZ6Zvb-tYflroyv4qG1iVeHQP5CvYxoOfD5tyrU1GRxTxtkUm50K2UUkqB6vlGH1BFvTRaFsX5a9TqJZVeWKL6wNmsgeLdmlNXsnXflUmff34T9fG_177G3jtqCYZNXtjH7ZMfgB7rYwDcV_1W7i7QrPxzdVjkiIjo6oskMialKg8JXNlu5aSJdrrn2ptyArXfk9uc7JR_SBfXK3VKRmRZbX-YR7qqZseJKRtfnII15Px6mzqOREGTzPJSk9SprOQRyZVwrcyEtwMldDIuhTVjOpEa2QZiO9Q6VQbahhNskSmIfUTkUaZfwSdvMjNOyDZEJ9HPDHMjwIuqNJSKGbQwYwyG83rwqAFJdauQ7kVyriLa0-FyhhhjC2MsYOxC59_z_jedOd4YeyhReWPcQ0gXei1wMfu-72PfYq00RaZi_fPTDuGnelqPotn54vLD7Br39TkBfagU64r8xG5Spl8qnfoL0xH45Q
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Optimization+of+Automated+and+Manual+Software+Tests+in+Industrial+Practice%3A+A+Survey+and+Historical+Analysis&rft.jtitle=IEEE+transactions+on+software+engineering&rft.au=Haas%2C+Roman&rft.au=N%C3%B6mmer%2C+Raphael&rft.au=Juergens%2C+Elmar&rft.au=Apel%2C+Sven&rft.date=2024-08-01&rft.issn=0098-5589&rft.eissn=1939-3520&rft.volume=50&rft.issue=8&rft.spage=2005&rft.epage=2020&rft_id=info:doi/10.1109%2FTSE.2024.3418191&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TSE_2024_3418191
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0098-5589&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0098-5589&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0098-5589&client=summon