Automatically Documenting Unit Test Cases

• Published in: 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST) pp. 341 - 352
• Main Authors: Boyang Li, Vendome, Christopher, Linares-Vasquez, Mario, Poshyvanyk, Denys, Kraft, Nicholas A.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.04.2016
• Subjects: Documentation; Electronic mail; History; Maintenance engineering; natural language processing; Open source software; Software systems; summarization; unit test
• DOI: 10.1109/ICST.2016.30

Maintaining unit test cases is important during the maintenance and evolution of a software system. In particular, automatically documenting these unit test cases can ameliorate the burden on developers maintaining them. For instance, by relying on up-to-date documentation, developers can more easily identify test cases that relate to some new or modified functionality of the system. We surveyed 212 developers (both industrial and open-source) to understand their perspective towards writing, maintaining, and documenting unit test cases. In addition, we mined change histories of C# software systems and empirically found that unit test methods seldom had preceding comments and infrequently had inner comments, and both were rarely modified as those methods were modified. In order to support developers in maintaining unit test cases, we propose a novel approach - UnitTestScribe - that combines static analysis, natural language processing, backward slicing, and code summarization techniques to automatically generate natural language documentation of unit test cases. We evaluated UnitTestScribe on four subject systems by means of an online survey with industrial developers and graduate students. In general, participants indicated that UnitTestScribe descriptions are complete, concise, and easy to read.