An Innovative, No-cost, Evidence-Based Smartphone Platform for Resident Evaluation

• Published in: Journal of surgical education Vol. 73; no. 6; pp. e14 - e18
• Main Author: Green, John M., MD
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.11.2016
• Subjects: Adult; Clinical Competence; competency; Cost Savings; Education, Medical, Graduate - methods; Educational Measurement - methods; evaluation; Evidence-Based Medicine; feedback; Female; General Surgery - education; Humans; Internship and Residency - methods; Interpersonal and Communication Skills; Male; Medical Knowledge; non-technical skills; North Carolina; Observational Studies as Topic; Practice-Based Learning and Improvement; Professionalism; Prospective Studies; smartphone; Smartphone - economics; Smartphone - utilization; Surgery; evaluation; timely feedback; competency; Medical Knowledge; technical and non-technical evaluation; Practice-Based Learning and Improvement; Professionalism; Interpersonal and Communication Skills; feedback; smartphone; non-technical skills
• ISSN: 1931-7204; 1878-7452
• DOI: 10.1016/j.jsurg.2016.07.016

Purpose Timely performance evaluation and feedback are critical to resident development. However, formulating and delivering this information disrupts physician workflow, leading to low participation. This study was designed to determine if a locally developed smartphone platform would integrate regular evaluation into daily processes and thus increase faculty participation in timely resident evaluation. Methods Formal, documented resident operative and patient interaction evaluations were compiled over an 8-month study period. The study was divided into two 4-month phases. No changes to the existing evaluation methods were made during Phase 1. Phase 2 began after a washout period of 2 weeks and coincided with the launch of a smartphone-based platform. The platform uses a combination of Likert scale questions and the Dreyfus model of skill acquisition to describe competence levels in technical and nontechnical skills. The instrument inflicts minimal effect on surgeon workflow, with the aim of integrating resident evaluation into daily processes. The number of different faculty members performing evaluations, resident level (postgraduate year), type of interaction or procedure, and competency data were compiled. All evaluations were tracked by the program director as they were automatically uploaded into a database. Faculty members were introduced to the new platform at the beginning of Phase 2, and previous methods of evaluation continued to be encouraged and were considered valid throughout both phases of the study. Data were analyzed using Fisher exact test for specific PGY level, and chi-square test was used for overall program analysis. Statistical significance was set at p < 0.05. Results Total faculty engagement, that is, number of faculty members completing evaluations, increased from 13% (5/38) in Phase 1 to 53% (20/38) in Phase 2. During Phase 1, all evaluations consisted of online forms through the department’s established system or e-mails to the program director. Evaluations were completed in 0.9% (15/1599) of cases residents completed in Phase 1 versus 12% (217/1812) of those in Phase 2. During Phase 2, evaluations were conducted exclusively using the new platform. This was done based on participant’s choice. Total numbers of residents and core faculty members did not change between Phases 1 and 2. Conclusions A smartphone-based platform can be created with existing technology at no cost. It is adaptable and can be updated in real-time and can employ validated scales to build an evaluation portfolio for learners assessing technical and nontechnical skills. Furthermore, and perhaps most importantly, it can be designed to integrate into existing workflow patterns to increase faculty participation.