Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical study

• Published in: International Journal of Educational Technology in Higher Education Vol. 21; no. 1; pp. 14 - 22
• Main Authors: Sun, Dan, Boudouaia, Azzeddine, Zhu, Chengcong, Li, Yan
• Format: Journal Article
• Language: English
• Published: Cham Springer International Publishing 01.12.2024; Springer Nature B.V; SpringerOpen
• Subjects: Artificial intelligence; Behavioral analysis; Chatbots; ChatGPT; College student; College students; Colleges & universities; Computer Appl. in Social and Behavioral Sciences; Computer Science; Computers and Education; Copying; Data analysis; Debugging; Design of experiments; Educational Technology; Empirical analysis; Feedback; Higher Education; Humanities; Information Systems Applications (incl.Internet); Instructional design; Law; Messages; Multidimensional data; Multidimensional methods; New advances in artificial intelligence applications in higher education; Perception; Perceptions; Programming; Programming learning; Quasiexperimental Design; Research Article; Research Design; Statistics for Social Sciences; Students; Troubleshooting; User experience; Programming learning; College student; Perception; ChatGPT; Behavioral analysis
• ISSN: 2365-9440; 2365-9440
• DOI: 10.1186/s41239-024-00446-5

ChatGPT, an AI-based chatbot with automatic code generation abilities, has shown its promise in improving the quality of programming education by providing learners with opportunities to better understand the principles of programming. However, limited empirical studies have explored the impact of ChatGPT on learners’ programming processes. This study employed a quasi-experimental design to explore the possible impact of ChatGPT-facilitated programming mode on college students’ programming behaviors, performances, and perceptions. 82 college students were randomly divided into two classes. One class employed ChatGPT-facilitated programming (CFP) practice and the other class utilized self-directed programming (SDP) mode. Mixed methods were utilized to collect multidimensional data. Data analysis uncovered some intriguing results. Firstly, students in the CFP mode had more frequent behaviors of debugging and receiving error messages, as well as pasting console messages on the website and reading feedback. At the same time, students in the CFP mode had more frequent behaviors of copying and pasting codes from ChatGPT and debugging, as well as pasting codes to ChatGPT and reading feedback from ChatGPT. Secondly, CFP practice would improve college students’ programming performance, while the results indicated that there was no statistically significant difference between the students in CFP mode and the SDP mode. Thirdly, student interviews revealed three highly concerned themes from students' user experience about ChatGPT: the services offered by ChatGPT, the stages of ChatGPT usage, and experience with ChatGPT. Finally, college students’ perceptions toward ChatGPT significantly changed after CFP practice, including its perceived usefulness, perceived ease of use, and intention to use. Based on these findings, the study proposes implications for future instructional design and the development of AI-powered tools like ChatGPT.