Measuring the Runtime Performance of C++ Code Written by Humans Using Github Copilot

GitHub Copilot is an artificially intelligent programming assistant used by many developers. While a few studies have evaluated the security risks of using Copilot, there has not been any study to show if it aids developers in producing code with better runtime performance. We evaluate the runtime p...

Full description

Saved in:
Bibliographic Details
Published inProceedings / International Conference on Software Engineering pp. 2062 - 2074
Main Authors Erhabor, Daniel, Udayashankar, Sreeharsha, Nagappan, Meiyappan, Al-Kiswany, Samer
Format Conference Proceeding
LanguageEnglish
Published IEEE 26.04.2025
Subjects
Atmospheric measurements
C++ languages
code completion
code generation
Codes
copilot
language models
Particle measurements
Programming
programming assistants
Runtime
Security
Software development management
Software measurement
Online AccessGet full text

Cover

More Information
Summary:GitHub Copilot is an artificially intelligent programming assistant used by many developers. While a few studies have evaluated the security risks of using Copilot, there has not been any study to show if it aids developers in producing code with better runtime performance. We evaluate the runtime performance of C++ code produced when developers use GitHub Copilot versus when they do not. To this end, we conducted a user study with 32 participants where each participant solved two C++ programming problems, one with Copilot and the other without it and measured the runtime performance of the participants' solutions on our test data. Our results suggest that using Copilot may produce \mathbf{C + +} code with (statistically significant) slower runtime performance.
ISSN:1558-1225
DOI:10.1109/ICSE55347.2025.00059