Semantic Robustness of Models of Source Code

• Published in: 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) pp. 526 - 537
• Main Authors: Henkel, Jordan, Ramakrishnan, Goutham, Wang, Zi, Albarghouthi, Aws, Jha, Somesh, Reps, Thomas
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.03.2022
• Subjects: Codes; Data models; Documentation; Learning from source code; Neural networks; Perturbation methods; Robust training; Semantics; Semantics preserving transformations; Training
• DOI: 10.1109/SANER53432.2022.00070

Deep neural networks are vulnerable to adversarial examples-small input perturbations that result in incorrect predictions. We study this problem for models of source code, where we want the neural network to be robust to source-code modifications that preserve code functionality. To facilitate training robust models, we define a powerful and generic adversary that can employ sequences of parametric, semantics-preserving program transformations. We then explore how, with such an adversary, one can train models that are robust to adversarial program transformations. We conduct a thorough evaluation of our approach and find several surprising facts: we find robust training to beat dataset augmentation in every evaluation we performed; we find that a state-of-the-art architecture (code2seq) for models of code is harder to make robust than a simpler baseline; additionally, we find code2seq to have surprising weaknesses not present in our simpler baseline model; finally, we find that robust models perform better against unseen data from different sources (as one might hope)-however, we also find that robust models are not clearly better in the cross-language transfer task. To the best of our knowledge, we are the first to study the interplay between robustness of models of code and the domain-adaptation and cross-language- transfer tasks.