Automatic item generation: foundations and machine learning-based approaches for assessments

This mini review summarizes the current state of knowledge about automatic item generation in the context of educational assessment and discusses key points in the item generation pipeline. Assessment is critical in all learning systems and digitalized assessments have shown significant growth over...

Full description

Saved in:
Bibliographic Details
Published inFrontiers in education (Lausanne) Vol. 8
Main Authors Circi, Ruhan, Hicks, Juanita, Sikali, Emmanuel
Format Journal Article
LanguageEnglish
Published Frontiers Media S.A 10.05.2023
Subjects
automatic item generation
digital assessments
item models
machine learning approaches, NLP
Online AccessGet full text

Cover

Abstract This mini review summarizes the current state of knowledge about automatic item generation in the context of educational assessment and discusses key points in the item generation pipeline. Assessment is critical in all learning systems and digitalized assessments have shown significant growth over the last decade. This leads to an urgent need to generate more items in a fast and efficient manner. Continuous improvements in computational power and advancements in methodological approaches, specifically in the field of natural language processing, provide new opportunities as well as new challenges in automatic generation of items for educational assessment. This mini review asserts the need for more work across a wide variety of areas for the scaled implementation of AIG.
AbstractList This mini review summarizes the current state of knowledge about automatic item generation in the context of educational assessment and discusses key points in the item generation pipeline. Assessment is critical in all learning systems and digitalized assessments have shown significant growth over the last decade. This leads to an urgent need to generate more items in a fast and efficient manner. Continuous improvements in computational power and advancements in methodological approaches, specifically in the field of natural language processing, provide new opportunities as well as new challenges in automatic generation of items for educational assessment. This mini review asserts the need for more work across a wide variety of areas for the scaled implementation of AIG.
Author Circi, Ruhan
Hicks, Juanita
Sikali, Emmanuel
Author_xml – sequence: 1
  givenname: Ruhan
  surname: Circi
  fullname: Circi, Ruhan
– sequence: 2
  givenname: Juanita
  surname: Hicks
  fullname: Hicks, Juanita
– sequence: 3
  givenname: Emmanuel
  surname: Sikali
  fullname: Sikali, Emmanuel
BookMark eNpNkE1LAzEQhoMoWGt_gLf9A1vztcmut1L8KBS89OBBCLPJpG7pJiXZHvz3blsRT_PODPPAPHfkOsSAhDwwOheibh49uqOdc8rFvK5qrsUVmfCKypLX8uP6X74ls5x3lFJOa6VVMyGfi-MQexg6W3QD9sUWA6axjeGp8PEY3DnnAoIrerBfXcBij5BCF7ZlCxldAYdDiuMK83iRCsgZc-4xDPme3HjYZ5z91inZvDxvlm_l-v11tVysSysqOZRKU46A2gnmmopJWjtspNIauNatE7L1lFvqnRMAbQXMKtCUedtUFWgmpmR1wboIO3NIXQ_p20TozHkQ09ZAGj_co2GCo9PQtJZLCc42Ap2vQLVMolXKjix2YdkUc07o_3iMmpNsc5ZtTrLNRbb4AU_Id9E
CitedBy_id crossref_primary_10_12793_tcp_2023_31_e16
crossref_primary_10_1186_s12909_024_05250_3
crossref_primary_10_1186_s40468_024_00291_3
Cites_doi 10.1080/08957347.2016.1171768
10.1017/S1351324921000139
10.1177/0146621617726788
10.4324/9780203850381
10.1111/jedm.12166
10.1108/IJILT-09-2017-0085
10.1007/978-3-319-93843-1_2
10.3109/0142159X.2016.1150989
10.4324/9780203803912
10.1007/978-3-642-32600-4_13
10.1186/s41039-020-00134-8
10.1016/j.lindif.2011.11.005
10.1111/j.1365-2923.1986.tb01042.x
10.1111/emip.12018
10.1609/aaai.v33i01.33016602
10.1007/s11336-021-09823-9
10.1007/s11336-018-9608-y
10.1007/S11336-011-9204-X
10.7333/1608-040101
10.3389/fpsyg.2018.01286
10.1007/s40593-019-00186-y
10.4324/9781003025634
10.1177/001316447803800104
10.1080/10401334.2016.1146608
10.5815/ijmecs.2015.01.01
10.3389/frai.2022.903077
10.1111/j.1365-2923.2012.04289.x
ContentType Journal Article
DBID AAYXX
CITATION
DOA
DOI 10.3389/feduc.2023.858273
DatabaseName CrossRef
DOAJ Directory of Open Access Journals
DatabaseTitle CrossRef
DatabaseTitleList CrossRef
Database_xml – sequence: 1
  dbid: DOA
  name: Directory of Open Access Journals
  url: https://www.doaj.org/
  sourceTypes: Open Website
DeliveryMethod fulltext_linktorsrc
Discipline Education
EISSN 2504-284X
ExternalDocumentID oai_doaj_org_article_132ed7a9bc244adc93edf5a6b14ec66c
10_3389_feduc_2023_858273
GroupedDBID 9T4
AAFWJ
AAYXX
ACXDI
ADBBV
AFPKN
ALMA_UNASSIGNED_HOLDINGS
BCNDV
CITATION
GROUPED_DOAJ
M~E
OK1
ID FETCH-LOGICAL-c354t-6702eae7d31d951408de94677a277bd34bf02c0fdd3aab5a1c6a701fc955a713
IEDL.DBID DOA
ISSN 2504-284X
IngestDate Tue Oct 22 15:02:48 EDT 2024
Fri Aug 23 01:33:16 EDT 2024
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c354t-6702eae7d31d951408de94677a277bd34bf02c0fdd3aab5a1c6a701fc955a713
OpenAccessLink https://doaj.org/article/132ed7a9bc244adc93edf5a6b14ec66c
ParticipantIDs doaj_primary_oai_doaj_org_article_132ed7a9bc244adc93edf5a6b14ec66c
crossref_primary_10_3389_feduc_2023_858273
PublicationCentury 2000
PublicationDate 2023-05-10
PublicationDateYYYYMMDD 2023-05-10
PublicationDate_xml – month: 05
  year: 2023
  text: 2023-05-10
  day: 10
PublicationDecade 2020
PublicationTitle Frontiers in education (Lausanne)
PublicationYear 2023
Publisher Frontiers Media S.A
Publisher_xml – name: Frontiers Media S.A
References Kim (ref32) 2019; 33
Gierl (ref25) 2012; 46
Fridenfalk (ref18) 2013
Latifi (ref37) 2013
Hommel (ref29) 2022; 87
Arendasy (ref1) 2012; 22
Gierl (ref21) 2013; 32
Pan (ref38) 2019
Attali (ref3) 2018
Sun (ref43) 2018
Zhou (ref49) 2019
Wang (ref46) 2018
Embretson (ref16) 2018; 55
Colvin (ref13) 2016; 4
Gierl (ref23) 2016; 29
Roid (ref42) 1978; 38
Bormuth (ref10) 1969
von Davier (ref44) 2018; 83
LaDuca (ref35) 1986; 20
Pugh (ref39) 2016; 38
Harrison (ref28) 2021
Fattoh (ref17) 2015; 7
Bejar (ref6) 2003; 2
Pugh (ref40) 2020; 15
Khodeir (ref31) 2018; 35
Baghaee (ref5) 2017
Drasgow (ref14) 2006
Rajpurkar (ref41) 2016
Wang (ref47) 2018
Gierl (ref26) 2008; 7
Kaliski (ref30) 2020
Attali (ref4) 2022; 5
Geerlings (ref19) 2011; 76
Lai (ref36) 2016; 28
Gierl (ref24) 2021
Kurdi (ref34) 2019; 30
von Davier (ref45) 2019
Blum (ref8) 2018; 9
Bo (ref9) 2020
Zhou (ref50) 2017
Gierl (ref20) 2012
Zhao (ref48) 2018
Haladyna (ref27) 2013
Gierl (ref22) 2018; 42
Atapattu (ref2) 2012
Du (ref15) 2017
Blšták (ref7) 2022; 28
Kumar (ref33) 2019
Chan (ref12) 2019
Burke (ref11) 2020
References_xml – volume: 29
  start-page: 196
  year: 2016
  ident: ref23
  article-title: Evaluating the psychometric characteristics of generated multiple-choice test items
  publication-title: Appl. Meas. Educ.
  doi: 10.1080/08957347.2016.1171768
  contributor:
    fullname: Gierl
– year: 2020
  ident: ref9
  contributor:
    fullname: Bo
– year: 2019
  ident: ref12
  contributor:
    fullname: Chan
– year: 2016
  ident: ref41
  contributor:
    fullname: Rajpurkar
– volume: 28
  start-page: 487
  year: 2022
  ident: ref7
  article-title: Automatic question generation based on sentence structure analysis using machine learning approach
  publication-title: Nat. Lang. Eng.
  doi: 10.1017/S1351324921000139
  contributor:
    fullname: Blšták
– volume: 42
  start-page: 42
  year: 2018
  ident: ref22
  article-title: Using automatic item generation to create solutions and rationales for computerized formative testing
  publication-title: Appl. Psychol. Meas.
  doi: 10.1177/0146621617726788
  contributor:
    fullname: Gierl
– year: 2021
  ident: ref28
  contributor:
    fullname: Harrison
– start-page: 17
  volume-title: Developing and Validating Test Items
  year: 2013
  ident: ref27
  article-title: Developing the test item
  doi: 10.4324/9780203850381
  contributor:
    fullname: Haladyna
– volume: 55
  start-page: 112
  year: 2018
  ident: ref16
  article-title: Automatic item generation: a more efficient process for developing mathematics achievement items?
  publication-title: J. Educ. Meas.
  doi: 10.1111/jedm.12166
  contributor:
    fullname: Embretson
– volume: 35
  start-page: 199
  year: 2018
  ident: ref31
  article-title: Generating story problems via controlled parameters in a web-based intelligent tutoring system
  publication-title: Int. J. Inf. Learn. Technol.
  doi: 10.1108/IJILT-09-2017-0085
  contributor:
    fullname: Khodeir
– year: 2020
  ident: ref30
  contributor:
    fullname: Kaliski
– volume: 2
  start-page: 1
  year: 2003
  ident: ref6
  article-title: A feasibility study of on-the-fly item generation in adaptive testing
  publication-title: J. Technol. Learn. Assess.
  contributor:
    fullname: Bejar
– year: 2013
  ident: ref37
  contributor:
    fullname: Latifi
– year: 2018
  ident: ref46
  contributor:
    fullname: Wang
– start-page: 17
  volume-title: Artificial intelligence in education: 19th International Conference
  year: 2018
  ident: ref3
  article-title: Automatic item generation unleashed: an evaluation of a large-scale deployment of item models
  doi: 10.1007/978-3-319-93843-1_2
  contributor:
    fullname: Attali
– volume: 38
  start-page: 838
  year: 2016
  ident: ref39
  article-title: Using cognitive models to develop quality multiple-choice questions
  publication-title: Med. Teach.
  doi: 10.3109/0142159X.2016.1150989
  contributor:
    fullname: Pugh
– volume-title: Automatic Item Generation: Theory and Practice
  year: 2012
  ident: ref20
  doi: 10.4324/9780203803912
  contributor:
    fullname: Gierl
– volume-title: Database and Expert Systems Applications. DEXA 2012
  year: 2012
  ident: ref2
  article-title: Automated extraction of semantic concepts from semi structured data: supporting computer-based education through the analysis of lecture notes
  doi: 10.1007/978-3-642-32600-4_13
  contributor:
    fullname: Atapattu
– volume: 15
  start-page: 1
  year: 2020
  ident: ref40
  article-title: Can automated item generation be used to develop high quality MCQs that assess application of knowledge?
  publication-title: Res. Pract. Technol. Enhanc. Learn.
  doi: 10.1186/s41039-020-00134-8
  contributor:
    fullname: Pugh
– volume-title: On a Theory of Achievement Test Items
  year: 1969
  ident: ref10
  contributor:
    fullname: Bormuth
– year: 2017
  ident: ref5
  contributor:
    fullname: Baghaee
– volume: 22
  start-page: 112
  year: 2012
  ident: ref1
  article-title: Using automatic item generation to meet the increasing item demands of high-stakes assessment
  publication-title: Learn. Individ. Differ.
  doi: 10.1016/j.lindif.2011.11.005
  contributor:
    fullname: Arendasy
– year: 2018
  ident: ref43
  contributor:
    fullname: Sun
– volume: 20
  start-page: 53
  year: 1986
  ident: ref35
  article-title: Item modeling procedures for constructing content-equivalent multiple-choice questions
  publication-title: Med. Educ.
  doi: 10.1111/j.1365-2923.1986.tb01042.x
  contributor:
    fullname: LaDuca
– year: 2017
  ident: ref50
  contributor:
    fullname: Zhou
– year: 2018
  ident: ref47
  contributor:
    fullname: Wang
– volume: 32
  start-page: 36
  year: 2013
  ident: ref21
  article-title: Instructional topics in educational measurement (items) module: using automated processes to generate test items
  publication-title: Educ. Meas. Issues Pract.
  doi: 10.1111/emip.12018
  contributor:
    fullname: Gierl
– volume: 33
  start-page: 6602
  year: 2019
  ident: ref32
  article-title: Improving neural question generation using answer separation
  publication-title: Proc. AAAI Conf. Artif. Intell.
  doi: 10.1609/aaai.v33i01.33016602
  contributor:
    fullname: Kim
– year: 2019
  ident: ref45
  contributor:
    fullname: von Davier
– year: 2019
  ident: ref38
  contributor:
    fullname: Pan
– year: 2019
  ident: ref49
  contributor:
    fullname: Zhou
– volume: 87
  start-page: 749
  year: 2022
  ident: ref29
  article-title: Transformer-based deep neural language modeling for construct-specific automatic item generation
  publication-title: Psychometrika
  doi: 10.1007/s11336-021-09823-9
  contributor:
    fullname: Hommel
– volume: 83
  start-page: 847
  year: 2018
  ident: ref44
  article-title: Automated item generation with recurrent neural networks
  publication-title: Psychometrika
  doi: 10.1007/s11336-018-9608-y
  contributor:
    fullname: von Davier
– volume: 76
  start-page: 337
  year: 2011
  ident: ref19
  article-title: Modeling rule-based item generation
  publication-title: Psychometrika
  doi: 10.1007/S11336-011-9204-X
  contributor:
    fullname: Geerlings
– year: 2018
  ident: ref48
  contributor:
    fullname: Zhao
– volume: 4
  start-page: 1
  year: 2016
  ident: ref13
  article-title: Effect of imprecise parameter estimation on ability estimation in a multistage test in an automatic item generation context
  publication-title: J. Comput. Adapt. Test.
  doi: 10.7333/1608-040101
  contributor:
    fullname: Colvin
– volume: 9
  start-page: 1
  year: 2018
  ident: ref8
  article-title: Automatic generation of figural analogies with the IMak package
  publication-title: Front. Psychol.
  doi: 10.3389/fpsyg.2018.01286
  contributor:
    fullname: Blum
– volume: 30
  start-page: 121
  year: 2019
  ident: ref34
  article-title: A systematic review of automatic question generation for educational purposes
  publication-title: Int. J. Artif. Intell. Educ.
  doi: 10.1007/s40593-019-00186-y
  contributor:
    fullname: Kurdi
– year: 2017
  ident: ref15
  contributor:
    fullname: Du
– volume: 7
  start-page: 1
  year: 2008
  ident: ref26
  article-title: Developing a taxonomy of item model types to promote assessment engineering
  publication-title: J. Technol. Learn. Assess.
  contributor:
    fullname: Gierl
– volume-title: Advanced Methods in Automatic Item Generation
  year: 2021
  ident: ref24
  doi: 10.4324/9781003025634
  contributor:
    fullname: Gierl
– year: 2020
  ident: ref11
  contributor:
    fullname: Burke
– volume: 38
  start-page: 19
  year: 1978
  ident: ref42
  article-title: A comparison of objective-based and modified-Bormuth item writing techniques
  publication-title: Educ. Psychol. Meas.
  doi: 10.1177/001316447803800104
  contributor:
    fullname: Roid
– start-page: 471
  volume-title: Educational Measurement
  year: 2006
  ident: ref14
  article-title: Technology and testing
  contributor:
    fullname: Drasgow
– year: 2019
  ident: ref33
  contributor:
    fullname: Kumar
– volume: 28
  start-page: 166
  year: 2016
  ident: ref36
  article-title: Using automatic item generation to improve the quality of MCQ distractors
  publication-title: Teach. Learn. Med.
  doi: 10.1080/10401334.2016.1146608
  contributor:
    fullname: Lai
– volume: 7
  start-page: 1
  year: 2015
  ident: ref17
  article-title: Semantic question generation using artificial immunity
  publication-title: Int. J. Mod. Educ. Comput. Sci.
  doi: 10.5815/ijmecs.2015.01.01
  contributor:
    fullname: Fattoh
– volume: 5
  start-page: 903077
  year: 2022
  ident: ref4
  article-title: The interactive reading task: transformer-based automatic item generation
  publication-title: Front. Artif. Intell.
  doi: 10.3389/frai.2022.903077
  contributor:
    fullname: Attali
– start-page: 365
  year: 2013
  ident: ref18
  article-title: System for automatic generation of examination papers in discrete mathematics
  publication-title: Proceedings of IADIS International Conference on e-Learning 2013 IADIS Multi Conference on Computer Science and Information Systems
  contributor:
    fullname: Fridenfalk
– volume: 46
  start-page: 757
  year: 2012
  ident: ref25
  article-title: Using automatic item generation to create multiple-choice test items
  publication-title: Med. Educ. J.
  doi: 10.1111/j.1365-2923.2012.04289.x
  contributor:
    fullname: Gierl
SSID ssj0002086769
Score 2.30252
Snippet This mini review summarizes the current state of knowledge about automatic item generation in the context of educational assessment and discusses key points in...
SourceID doaj
crossref
SourceType Open Website
Aggregation Database
SubjectTerms automatic item generation
digital assessments
item models
machine learning approaches, NLP
Title Automatic item generation: foundations and machine learning-based approaches for assessments
URI https://doaj.org/article/132ed7a9bc244adc93edf5a6b14ec66c
Volume 8
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1NSwMxEA3SkxfxE-sXOXgS0m42yWbjrYqlCHqq0IOwzCaTgmAr7fb_m2TXUk9evO5HCG-WzLzN5D1CbsHr0qPjTNtSMSllzsCgCSxFoOHGy5rHs8Mvr8XkTT7P1GzH6iv2hLXywC1ww8CW0GkwtQ2JCJw1Ap1XUNRcoi0Km1bfzOyQqY-0vVbG3s12GzOwMDP0UQl1EM3CB6Uqcy1-JaIdvf6UWMaH5KCrCOmonckR2cPFcTRT7hovTsj7aNMsk7IqjX9a6TwpRcd799RvXZHWFBaOfqbeSKSdGcScxSzl6I9yOK7DGysKWznO9SmZjp-mjxPWmSIwK5RsWKGzHAG1E9yF6khmpUMTVjsNuda1E7L2WW4z75wAqBVwW4DOuLdGKQiM9Iz0FssFnhMqNag8s8LLwMFAIeSOyzBCYCC21E71yd0PQNVXK31RBcoQ0awSmlVEs2rR7JOHCOH2wahanS6EWFZdLKu_YnnxH4Nckv04L5YUVq9Ir1lt8DoUDk19k76Rb2njxJM
link.rule.ids 315,783,787,867,2109,27936,27937
linkProvider Directory of Open Access Journals
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Automatic+item+generation%3A+foundations+and+machine+learning-based+approaches+for+assessments&rft.jtitle=Frontiers+in+education+%28Lausanne%29&rft.au=Circi%2C+Ruhan&rft.au=Hicks%2C+Juanita&rft.au=Sikali%2C+Emmanuel&rft.date=2023-05-10&rft.issn=2504-284X&rft.eissn=2504-284X&rft.volume=8&rft_id=info:doi/10.3389%2Ffeduc.2023.858273&rft.externalDBID=n%2Fa&rft.externalDocID=10_3389_feduc_2023_858273
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2504-284X&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2504-284X&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2504-284X&client=summon