Evaluating and Improving Unified Debugging

• Published in: IEEE transactions on software engineering Vol. 48; no. 11; pp. 4692 - 4716
• Main Authors: Benton, Samuel, Li, Xia, Lou, Yiling, Zhang, Lingming
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.11.2022; IEEE Computer Society
• Subjects: Automated program repair; Automation; Codes; Computer bugs; Debugging; fault localization; Fault location; Learning; Localization; Location awareness; Maintenance engineering; Manuals; Patches (structures); Software systems; unified debugging
• ISSN: 0098-5589; 1939-3520
• DOI: 10.1109/TSE.2021.3125203

Automated debugging techniques, including fault localization and program repair, have been studied for over a decade. However, the only existing connection between fault localization and program repair is that fault localization computes the potential buggy elements for program repair to patch. Recently, a pioneering work, ProFL, explored the idea of unified debugging to unify fault localization and program repair in the other direction for the first time to boost both areas. More specifically, ProFL utilizes the patch execution results from one state-of-the-art repair system, PraPR, to help improve state-of-the-art fault localization. In this way, ProFL not only improves fault localization for manual repair , but also extends the application scope of automated repair to all possible bugs (not only the small ratio of bugs that repair systems can automatically fix). However, ProFL only considers one program repair system (i.e., PraPR), and it is not clear how other existing program repair systems based on different designs contribute to unified debugging. In this work, we perform an extensive study of the unified debugging approach on 16 state-of-the-art program repair systems for the first time. Our initial experimental results on the widely studied Defects4J benchmark suite reveal various practical guidelines for unified debugging, such as (1) nearly all 16 studied repair systems positively contribute to unified debugging despite their varying repair capabilities, (2) repair systems targeting multi-edit patches can bring extraneous noise into unified debugging, (3) repair systems with more executed/plausible patches tend to perform better for unified debugging, (4) unified debugging effectiveness does not rely on the availability of correct patches from automated repair, and (5) we propose a new unified debugging technique, UniDebug++, which localizes over 20% more bugs within Top-1 than state-of-the-art unified debugging technique ProFL (evaluated against four Defects4J subjects). Furthermore, we conduct more comprehensive studies to extend the above experiments to make the following additional contributions: we (6) further perform an extensive study on 76.3% additional buggy versions from Defects4J (for Closure and Mockito) and confirm that UniDebug++ again outperforms ProFL by localizing 185 (out of 395 in total) bugs within Top-1, 14% more than ProFL, (7) investigate the impact of 33 SBFL formulae on unified debugging and observe that UniDebug++ consistently improves upon all formulae, e.g., 61% and 53% average improvement on MFR / MAR, (8) demonstrate that UniDebug++ can substantially boost state-of-the-art learning-based method-level fault localization techniques, (9) extend unified debugging to the statement level for first time and observe that UniDebug++ localizes 78 (out of 395 in total) bugs within Top-1 (22% more bugs than ProFL) and outperforms state-of-the-art learning-based fault localization techniques by 30%, and finally (10) propose a new technique, UniDebug+<inline-formula><tex-math notation="LaTeX">^\star</tex-math> <mml:math><mml:msup><mml:mrow/><mml:mi>★</mml:mi></mml:msup></mml:math><inline-graphic xlink:href="benton-ieq1-3125203.gif"/> </inline-formula>, based on detailed patch statistics, to improve upon UniDebug++, e.g., further localizing up to 9% more bugs within Top-1 than UniDebug++.