Application of a systematic approach to evaluating psychometric properties of a cumulative exit-from-degree objective structured clinical examination (OSCE)

• Published in: Currents in pharmacy teaching and learning Vol. 9; no. 6; pp. 1091 - 1098
• Main Authors: Sobh, Ahmed Hesham, Austin, Zubin, Izham M. I., Mohamed, Diab, Mohammad I., Wilby, Kyle John
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.11.2017
• Subjects: Assessment; Clinical Competence - standards; Educational Measurement - methods; Evaluation; Humans; OSCE; Pharmacy; Psychometrics; Psychometrics - instrumentation; Psychometrics - methods; Psychometrics - standards; Qatar; Reproducibility of Results; Students, Pharmacy - psychology; Systems Analysis; Universities - organization & administration; Evaluation; Psychometrics; OSCE; Pharmacy; Assessment
• ISSN: 1877-1297; 1877-1300
• DOI: 10.1016/j.cptl.2017.07.011

Objective structured clinical examinations (OSCEs) are considered gold standard performance-based assessments yet comprehensive evaluation data is currently lacking. The objective of this study was to critically evaluate the psychometric properties of a cumulative OSCE for graduating pharmacy students in Qatar for which policies and procedures were adapted from a Canadian context. A 10-station OSCE was conducted for graduating students in Qatar. Evaluation included assessment of pass rates, predictive validity, concurrent validity, internal validity, content validity, interrater reliability, and internal consistency. Twenty-six students completed the OSCE. Three stations achieved pass rates < 80%. Scores from professional skills and case-based learning courses, formative OSCEs, and cumulative grade point averages correlated with OSCE scores (p < 0.05). Average correlation between assessors’ analytical and global scoring was moderate (r = 0.52). Average interrater reliability was excellent for analytical scoring (ICC = 0.88) and moderate for global scoring (ICC = 0.61). Excellent internal consistency was demonstrated for overall performance (α = 0.927). Students generally agreed stations represented real practice scenarios (range per station, 30–100%). The evaluation model identified strengths and weaknesses in assessment and curricular considerations. The OSCE demonstrated acceptable validity and reliability as an adapted assessment.