Examining the impact of specific types of item-writing flaws on student performance and psychometric properties of the multiple choice question

Background: Item-writing flaws (IWFs) are common in multiple choice questions (MCQs) despite item-writing guidelines. Previous studies have shown that IWFs impact validity as observed through student performance, item difficulty, and discrimination. Most previous studies have examined IWFs collectiv...

Full description

Saved in:
Bibliographic Details
Published inMedEdPublish Vol. 7; no. 4
Main Authors Hannah Pham, James Besanko, Peter Devitt
Format Journal Article
LanguageEnglish
Published F1000 Research Ltd 01.10.2018
Subjects
Online AccessGet full text

Cover

Loading…
Abstract Background: Item-writing flaws (IWFs) are common in multiple choice questions (MCQs) despite item-writing guidelines. Previous studies have shown that IWFs impact validity as observed through student performance, item difficulty, and discrimination. Most previous studies have examined IWFs collectively and have shown that they have a diverse impact. The aim of the study was to determine if the effects of individual types of IWFs are systematic and predictable. Method: A cross-over study design was used. 100 pairs of MCQ items (with and without an IWF) were constructed to test 10 types of IWFs. Medical students were invited to participate in a mock examination. Paper A consisted of 50 flawed followed by 50 unflawed items. Paper B consisted of 50 unflawed followed by 50 flawed items. The effect of each of the IWFs on mean item scores, item difficulty and discrimination were examined. Results: The hypothesised effect of IWFs on mean item scores was confirmed in only 4 out of 10 cases. 'Longest choice is correct', 'Clues to the right answer (Eponymous terms)' and 'Implausible distractors' positively impacted, while 'Central idea in choices rather than stem' negatively impacted mean item scores. Other flaws had either the opposite or no statistically significant effect. IWFs did not impact item difficulty or discrimination. Conclusion: The effect of IWFs is neither systematic nor predictable. Unpredictability in assessment produces error and thus loss of validity. Therefore, IWFs should be avoided. Faculties should be encouraged to invest in item-writing workshops in order to improve MCQs. However, the cost of doing so should be carefully weighed against the benefits of developing programmes of assessment.
AbstractList Background: Item-writing flaws (IWFs) are common in multiple choice questions (MCQs) despite item-writing guidelines. Previous studies have shown that IWFs impact validity as observed through student performance, item difficulty, and discrimination. Most previous studies have examined IWFs collectively and have shown that they have a diverse impact. The aim of the study was to determine if the effects of individual types of IWFs are systematic and predictable. Method: A cross-over study design was used. 100 pairs of MCQ items (with and without an IWF) were constructed to test 10 types of IWFs. Medical students were invited to participate in a mock examination. Paper A consisted of 50 flawed followed by 50 unflawed items. Paper B consisted of 50 unflawed followed by 50 flawed items. The effect of each of the IWFs on mean item scores, item difficulty and discrimination were examined. Results: The hypothesised effect of IWFs on mean item scores was confirmed in only 4 out of 10 cases. 'Longest choice is correct', 'Clues to the right answer (Eponymous terms)' and 'Implausible distractors' positively impacted, while 'Central idea in choices rather than stem' negatively impacted mean item scores. Other flaws had either the opposite or no statistically significant effect. IWFs did not impact item difficulty or discrimination. Conclusion: The effect of IWFs is neither systematic nor predictable. Unpredictability in assessment produces error and thus loss of validity. Therefore, IWFs should be avoided. Faculties should be encouraged to invest in item-writing workshops in order to improve MCQs. However, the cost of doing so should be carefully weighed against the benefits of developing programmes of assessment.
Author James Besanko
Hannah Pham
Peter Devitt
Author_xml – sequence: 1
  fullname: Hannah Pham
  organization: University of Adelaide
– sequence: 2
  fullname: James Besanko
  organization: Royal Adelaide Hospital
– sequence: 3
  fullname: Peter Devitt
  organization: Royal Adelaide Hospital, University of Adelaide
BookMark eNqtjU1OwzAQhSNEFwV6B18gEqnTtF6jItizt6bOuB3kP-yJSk7BlXGAI7Aa6XvvfXPX3IYY8KZZb2W3bfdKDevm6_gJngKFs-ALCvIJDItoRUloyJIRPCcsCyFG314z8VK2Dq6VBlF4GjGwSJhtzB6CQQFhFKnM5hI9cq6OlGPNmX5FyyM_OabkUNQS1cnHhIUphodmZcEV3Pzd--b1-fj29NKOEd51yuQhzzoC6R8Q81lDFRuHulNW9uagsLOP_W6wJzmc7MGAVAr2ppfyP13f1khwYw
ContentType Journal Article
DBID DOA
DatabaseName Directory of Open Access Journals
DatabaseTitleList
Database_xml – sequence: 1
  dbid: DOA
  name: DOAJ Directory of Open Access Journals
  url: https://www.doaj.org/
  sourceTypes: Open Website
DeliveryMethod fulltext_linktorsrc
EISSN 2312-7996
ExternalDocumentID oai_doaj_org_article_19f34c89e1f0456fb36bf8ca399a7c43
GroupedDBID GROUPED_DOAJ
ID FETCH-doaj_primary_oai_doaj_org_article_19f34c89e1f0456fb36bf8ca399a7c433
IEDL.DBID DOA
IngestDate Tue Oct 22 15:14:01 EDT 2024
IsOpenAccess true
IsPeerReviewed false
IsScholarly false
Issue 4
Language English
LinkModel DirectLink
MergedId FETCHMERGED-doaj_primary_oai_doaj_org_article_19f34c89e1f0456fb36bf8ca399a7c433
OpenAccessLink https://doaj.org/article/19f34c89e1f0456fb36bf8ca399a7c43
ParticipantIDs doaj_primary_oai_doaj_org_article_19f34c89e1f0456fb36bf8ca399a7c43
PublicationCentury 2000
PublicationDate 2018-10-01
PublicationDateYYYYMMDD 2018-10-01
PublicationDate_xml – month: 10
  year: 2018
  text: 2018-10-01
  day: 01
PublicationDecade 2010
PublicationTitle MedEdPublish
PublicationYear 2018
Publisher F1000 Research Ltd
Publisher_xml – name: F1000 Research Ltd
Score 3.4251354
Snippet Background: Item-writing flaws (IWFs) are common in multiple choice questions (MCQs) despite item-writing guidelines. Previous studies have shown that IWFs...
SourceID doaj
SourceType Open Website
SubjectTerms Item difficulty
Item discrimination
Item writing flaws
Medical assessment
Multiple choice question
Psychometrics
Title Examining the impact of specific types of item-writing flaws on student performance and psychometric properties of the multiple choice question
URI https://doaj.org/article/19f34c89e1f0456fb36bf8ca399a7c43
Volume 7
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV09TwMxDLVQJxYEKojPygNrBHfJpclYUKvCwARSt1NyJBO9ntqi8i_4y9hJETAxwGoltuJIeY5lPwNcatM4760Vz8ZUghCqEpaAXSjjpXSeB9wmts8HPX1S97Nq9m3UF9eEZXrg7LirwkapGmNDETn6iF5qH8kAAasbNirzfF6X-TP1g3g_IcRkH_a2oR2OssoD2AltH97Hb26ehjAghVqYmxJxEZFbHLlMBzkJumIJp1HFhkmGaHF8cRuStrjK5JPYfVX4I_3-MbdPzXkgVoMdp9SXzI3KitjQZ6Eg0iJ6CzA9_3QHh3A3GT_eTgWfoO4y10TN7M9JQD6ptz6pf_OJPIJeu2jDMaC02ulyGKUNTgXlrC4b2hucd4oiIX8CN3-3d_ofSs5gl4KRzDVbnENvvXwNFwT4az9Id_sB7JG4EQ
link.rule.ids 314,780,784,2102
linkProvider Directory of Open Access Journals
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Examining+the+impact+of+specific+types+of+item-writing+flaws+on+student+performance+and+psychometric+properties+of+the+multiple+choice+question&rft.jtitle=MedEdPublish&rft.au=Hannah+Pham&rft.au=James+Besanko&rft.au=Peter+Devitt&rft.date=2018-10-01&rft.pub=F1000+Research+Ltd&rft.eissn=2312-7996&rft.volume=7&rft.issue=4&rft.externalDBID=DOA&rft.externalDocID=oai_doaj_org_article_19f34c89e1f0456fb36bf8ca399a7c43