Can automated item generation be used to develop high quality MCQs that assess application of knowledge?

The purpose of this study was to compare the quality of multiple choice questions (MCQs) developed using automated item generation (AIG) versus traditional methods, as judged by a panel of experts. The quality of MCQs developed using two methods (i.e., AIG or traditional) was evaluated by a panel of...

Full description

Saved in:
Bibliographic Details
Published inResearch and practice in technology enhanced learning Vol. 15; no. 1; pp. 1 - 13
Main Authors Pugh, Debra, De Champlain, André, Gierl, Mark, Lai, Hollis, Touchie, Claire
Format Journal Article
LanguageEnglish
Published Singapore Springer Singapore 05.06.2020
Springer
Asia-Pacific Society for Computers in Education (APSCE)
The Asia-Pacific Society for Computers in Education (APSCE)
Subjects
Automated item generation
Automation
Cognition & reasoning
Cognitive ability
Computer Assisted Testing
Education
Educational Technology
Evaluation
Learning and Instruction
Multiple Choice Tests
Multiple-choice questions
Quality
Quality assessment
Skills
Test Construction
Test Items
Thinking Skills
Automated item generation
Multiple-choice questions
Online AccessGet full text

Cover

More Information
Summary:The purpose of this study was to compare the quality of multiple choice questions (MCQs) developed using automated item generation (AIG) versus traditional methods, as judged by a panel of experts. The quality of MCQs developed using two methods (i.e., AIG or traditional) was evaluated by a panel of content experts in a blinded study. Participants rated a total of 102 MCQs using six quality metrics and made a judgment regarding whether or not each item tested recall or application of knowledge. A Wilcoxon two-sample test evaluated differences in each of the six quality metrics rating scales as well as an overall cognitive domain judgment. No significant differences were found in terms of item quality or cognitive domain assessed when comparing the two item development methods. The vast majority of items (> 90%) developed using both methods were deemed to be assessing higher-order skills. When compared to traditionally developed items, MCQs developed using AIG demonstrated comparable quality. Both modalities can produce items that assess higher-order cognitive skills.
ISSN:1793-7078
1793-2068
1793-7078
DOI:10.1186/s41039-020-00134-8