A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges

• Published in: Proceedings / International Conference on Software Engineering pp. 616 - 628
• Main Authors: Liang, Jenny T., Yang, Chenyang, Myers, Brad A.
• Format: Conference Proceeding
• Language: English
• Published: ACM 14.04.2024
• Subjects: AI programming assistants; Artificial intelligence; Computing methodologies → Natural language processing; Human-centered computing → Empirical studies in HCI; Programming; Software; Software and its engineering → Software notations and tools; Surveys; Syntactics; Task analysis; Usability; usability study
• DOI: 10.1145/3597503.3608128

The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools. To understand developers' practices while using these tools and the important usability challenges they face, we administered a survey to a large population of developers and received responses from a diverse set of 410 developers. Through a mix of qualitative and quantitative analyses, we found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax, but resonate less with using them to help brainstorm potential solutions. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements and because developers have trouble controlling the tool to generate the desired output. Our findings have implications for both creators and users of AI programming assistants, such as designing minimal cognitive effort interactions with these tools to reduce distractions for users while they are programming.