A Comparative Study of Class Rebalancing Methods for Security Bug Report Classification

• Published in: IEEE transactions on reliability Vol. 70; no. 4; pp. 1658 - 1670
• Main Authors: Zheng, Wei, Xun, Yuxing, Wu, Xiaoxue, Deng, Zhi, Chen, Xiang, Sui, Yulei
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.12.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Class imbalance problem; class rebalancing methods; Classification; classification algorithms; Comparative studies; Computer bugs; Debugging; Dimensionality reduction; Empirical analysis; Impact analysis; Performance enhancement; Prediction algorithms; Predictive models; Security; security bug report (SBRs) classification; Software algorithms; State-of-the-art reviews
• ISSN: 0018-9529; 1558-1721
• DOI: 10.1109/TR.2021.3118026

Identifying security bug reports (SBRs) accurately from a bug repository can reduce a software product's security risk. However, the class imbalance problem exists for SBR prediction since the number of SBRs is often limited, and this issue has not been thoroughly investigated in previous studies. In our study, we choose six real-world projects of different sizes with over 120 000 bug reports in total as our empirical subjects. We first analyze the impact of the class imbalance issue on SBR prediction and confirm its negative impact on prediction performance. Then we perform a comparative study of six state-of-the-art class rebalancing methods combined with five popular classification algorithms for SBR prediction. By comparing with the baseline method Farsec, using the class rebalancing methods can improve the performance in 78% of cases in the worst case. Moreover, the combination of the Rose and random forest classification algorithm can construct the model with the best performance, which increases the performance by 267% in the best case and 75% on average in terms of F1-score . Finally, we summarize eight main findings based on our empirical studies' results, which can provide guidelines for choosing appropriate class rebalancing methods and classifiers for SBR prediction in practice.