Framing our Expectations: Variability in Entrustable Professional Activity Assessments

• Published in: Journal of surgical education Vol. 81; no. 10; pp. 1355 - 1361
• Main Authors: Jones-Carr, Maggie E., McLeod, Chandler, Baker, Samantha, Lindeman, Brenessa
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.10.2024
• Subjects: Entrustable professional activities; frame-of-reference; learning assessments; resident evaluations; AA; Practice-Based Learning and Improvement (PBLI); Entrustable professional activities; PR; LP; Patient Care (PC); IS; learning assessments; DS; ABS; IHR; EPA; resident evaluations; frame-of-reference; Medical Knowledge (MK)
• ISSN: 1931-7204; 1878-7452; 1878-7452
• DOI: 10.1016/j.jsurg.2024.07.025

•Entrustable professional activities (EPAs) have been implemented as a framework for assessment of general surgery residents.•Higher inter-rater agreement between faculty and trainees was noted in evaluations of trainees with higher degrees of entrustment.•Faculty may be comparatively better than trainees at identifying the targeted level of entrustability based on observed behaviors.•Frame-of-reference training for trainees and faculty using video demonstrations may be useful during implementation of EPAs or for ongoing end-user development. To determine the ability of surgical trainees and faculty to correctly interpret entrustability of a resident learner in a modeled patient care scenario. Prospective study utilizing a web-based survey including 4 previously-recorded short videos of resident learners targeted to specific levels of the American Board of Surgery's (ABS) Entrustment Scale. Respondents were asked to choose the entrustment level that best corresponded to their observations of the learner in the video. Responses were subcategorized by low and high entrustment. Online, utilizing the Qualtrics survey platform. Survey targeting US surgical trainees and surgical faculty via email and social media. We received 31 complete responses and 2 responses which completed > 1 video assessment question without demographic information (n = 33). Respondents included 10 trainees (32%) and 21 attending surgeons (68%). Neither faculty nor trainees readily identified the targeted entrustment level for Question 1 (preoperative care of a patient with acute appendicitis with high entrustment, 36% correct), though evaluations of the remaining questions (2 through 4) demonstrated more accuracy (70, 84, and 75% correct, respectively). Faculty were more readily able than trainees to identify low entrustment (level Limited Participation) in intraoperative inguinal hernia repair (95% vs 60%, p = 0.03). After subcategorization to high and low entrustment, both residents and faculty were able to accurately identify entrustment 95% overall. Both trainees and attending surgeons were able to identify high- and low-performing residents on short video demonstrations using the ABS EPA entrustment scale. This provides additional evidence in support of the need for frequent observations of EPAs to account for the variability in raters’ perceptions in addition to complexity of clinical scenarios. Frame-of-reference training via a video-based platform may also be beneficial for both residents and faculty as an ongoing EPA implementation strategy.