A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges

The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related...

Full description

Saved in:
Bibliographic Details
Published inProceedings / International Conference on Software Engineering pp. 616 - 628
Main Authors Liang, Jenny T., Yang, Chenyang, Myers, Brad A.
Format Conference Proceeding
LanguageEnglish
Published ACM 14.04.2024
Subjects
AI programming assistants
Artificial intelligence
Computing methodologies → Natural language processing
Human-centered computing → Empirical studies in HCI
Programming
Software
Software and its engineering → Software notations and tools
Surveys
Syntactics
Task analysis
Usability
usability study
Online AccessGet full text

Cover

More Information
Summary:The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools. To understand developers' practices while using these tools and the important usability challenges they face, we administered a survey to a large population of developers and received responses from a diverse set of 410 developers. Through a mix of qualitative and quantitative analyses, we found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax, but resonate less with using them to help brainstorm potential solutions. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements and because developers have trouble controlling the tool to generate the desired output. Our findings have implications for both creators and users of AI programming assistants, such as designing minimal cognitive effort interactions with these tools to reduce distractions for users while they are programming.
ISSN:1558-1225
DOI:10.1145/3597503.3608128