A procedural skills OSCE: assessing technical and non-technical skills of internal medicine residents

• Published in: Advances in health sciences education : theory and practice Vol. 20; no. 1; pp. 85 - 100
• Main Authors: Pugh, Debra, Hamstra, Stanley J., Wood, Timothy J., Humphrey-Murto, Susan, Touchie, Claire, Yudkowsky, Rachel, Bordage, Georges
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer Netherlands 01.03.2015; Springer; Springer Nature B.V
• Subjects: Adult; Clinical Competence; Correlation; Education; Education, Medical, Graduate - standards; Educational Measurement - methods; Examiners; Female; Generalizability Theory; Graduate Students; Humans; Internal Medicine; Internal Medicine - education; Internship and Residency; Male; Medical Education; Medical Students; Models, Educational; Objective Tests; Ontario; Reproducibility of Results; Scores; Skills; Task Analysis; Test Reliability; Validity; Technical skills; Procedures; OSCE; Post-graduate trainees; Non-technical skills; Assessment
• ISSN: 1382-4996; 1573-1677
• DOI: 10.1007/s10459-014-9512-x

Internists are required to perform a number of procedures that require mastery of technical and non-technical skills, however, formal assessment of these skills is often lacking. The purpose of this study was to develop, implement, and gather validity evidence for a procedural skills objective structured clinical examination (PS-OSCE) for internal medicine (IM) residents to assess their technical and non-technical skills when performing procedures. Thirty-five first to third-year IM residents participated in a 5-station PS-OSCE, which combined partial task models, standardized patients, and allied health professionals. Formal blueprinting was performed and content experts were used to develop the cases and rating instruments. Examiners underwent a frame-of-reference training session to prepare them for their rater role. Scores were compared by levels of training, experience, and to evaluation data from a non-procedural OSCE (IM-OSCE). Reliability was calculated using Generalizability analyses. Reliabilities for the technical and non-technical scores were 0.68 and 0.76, respectively. Third-year residents scored significantly higher than first-year residents on the technical (73.5 vs. 62.2 %) and non-technical (83.2 vs. 75.1 %) components of the PS-OSCE ( p  < 0.05). Residents who had performed the procedures more frequently scored higher on three of the five stations ( p  < 0.05). There was a moderate disattenuated correlation (r = 0.77) between the IM-OSCE and the technical component of the PS-OSCE scores. The PS-OSCE is a feasible method for assessing multiple competencies related to performing procedures and this study provides validity evidence to support its use as an in-training examination.