To what extent could we detect field defects? An extended empirical study of false negatives in static bug-finding tools

Software defects can cause much loss. Static bug-finding tools are designed to detect and remove software defects and believed to be effective. However, do such tools in fact help prevent actual defects that occur in the field and reported by users? If these tools had been used, would they have dete...

Full description

Saved in:
Bibliographic Details
Published inAutomated software engineering Vol. 22; no. 4; pp. 561 - 602
Main Authors Thung, Ferdian, Lucia, Lo, David, Jiang, Lingxiao, Rahman, Foyzur, Devanbu, Premkumar T.
Format Journal Article
LanguageEnglish
Published New York Springer US 01.12.2015
Springer Nature B.V
Subjects
Artificial Intelligence
Computer Science
Defects
Effectiveness
Empirical analysis
Questions
Software
Software Engineering/Programming and Operating Systems
Source programs
Static bug-finding tools
Empirical study
False negatives
Online AccessGet full text

Cover

More Information
Summary:Software defects can cause much loss. Static bug-finding tools are designed to detect and remove software defects and believed to be effective. However, do such tools in fact help prevent actual defects that occur in the field and reported by users? If these tools had been used, would they have detected these field defects, and generated warnings that would direct programmers to fix them? To answer these questions, we perform an empirical study that investigates the effectiveness of five state-of-the-art static bug-finding tools (FindBugs, JLint, PMD, CheckStyle, and JCSC) on hundreds of reported and fixed defects extracted from three open source programs (Lucene, Rhino, and AspectJ). Our study addresses the question: To what extent could field defects be detected by state-of-the-art static bug-finding tools? Different from past studies that are concerned with the numbers of false positives produced by such tools, we address an orthogonal issue on the numbers of false negatives. We find that although many field defects could be detected by static bug-finding tools, a substantial proportion of defects could not be flagged. We also analyze the types of tool warnings that are more effective in finding field defects and characterize the types of missed defects. Furthermore, we analyze the effectiveness of the tools in finding field defects of various severities, difficulties, and types.
ISSN:0928-8910
1573-7535
DOI:10.1007/s10515-014-0169-8