Few-shot is enough: exploring ChatGPT prompt engineering method for automatic question generation in english education

• Published in: Education and information technologies Vol. 29; no. 9; pp. 11483 - 11515
• Main Authors: Lee, Unggi, Jung, Haewon, Jeon, Younghoon, Sohn, Younghoon, Hwang, Wonhee, Moon, Jewoong, Kim, Hyeoncheol
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.06.2024; Springer Nature B.V
• Subjects: Chatbots; Computer Appl. in Social and Behavioral Sciences; Computer Science; Computers and Education; Education; Educational Technology; Electronic Learning; Engineering; Generative artificial intelligence; Information Systems Applications (incl.Internet); Large language models; Teaching Methods; User Interfaces and Human Computer Interaction; Prompt Engineering; Automatic Question Generation; ChatGPT; Large Language Model; English Education
• ISSN: 1360-2357; 1573-7608
• DOI: 10.1007/s10639-023-12249-8

Through design and development research (DDR), we aimed to create a validated automatic question generation (AQG) system using large language models (LLMs) like ChatGPT, enhanced by prompting engineering techniques. While AQG has become increasingly integral to online learning for its efficiency in generating questions, issues such as inconsistent question quality and the absence of transparent and validated evaluation methods persist. Our research focused on creating a prompt engineering protocol tailored for AQG. This protocol underwent several iterations of refinement and validation to improve its performance. By gathering validation scores and qualitative feedback on the produced questions and the system’s framework, we examined the effectiveness of the system. The study findings indicate that our combined use of LLMs and prompt engineering in AQG produces questions with statistically significant validity. Our research further illuminates academic and design considerations for AQG design in English education: (a) certain question types might not be optimal for generation via ChatGPT, (b) ChatGPT sheds light on the potential for collaborative AI-teacher efforts in question generation, especially within English education.