Reading between the lines: faculty interpretations of narrative evaluation comments
Objectives Narrative comments are used routinely in many forms of rater‐based assessment. Interpretation can be difficult as a result of idiosyncratic writing styles and disconnects between literal and intended meanings. Our purpose was to explore how faculty attendings interpret and make sense of t...
Saved in:
Published in | Medical education Vol. 49; no. 3; pp. 296 - 306 |
---|---|
Main Authors | , , , |
Format | Journal Article |
Language | English |
Published |
England
Blackwell Publishing Ltd
01.03.2015
Wiley Subscription Services, Inc |
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Objectives
Narrative comments are used routinely in many forms of rater‐based assessment. Interpretation can be difficult as a result of idiosyncratic writing styles and disconnects between literal and intended meanings. Our purpose was to explore how faculty attendings interpret and make sense of the narrative comments on residents’ in‐training evaluation reports (ITERs) and to determine the language cues that appear to be influential in generating and justifying their interpretations.
Methods
A group of 24 internal medicine (IM) faculty attendings each categorised a subgroup of postgraduate year 1 (PGY1) and PGY2 IM residents based solely on ITER comments. They were then interviewed to determine how they had made their judgements. Constant comparative techniques from constructivist grounded theory were used to analyse the interviews and develop a framework to help in understanding how ITER language was interpreted.
Results
The overarching theme of ‘reading between the lines’ explained how participants read and interpreted ITER comments. Scanning for ‘flags’ was part of this strategy. Participants also described specific factors that shaped their judgements, including: consistency of comments; competency domain; specificity; quantity, and context (evaluator identity, rotation type and timing). There were several perceived purposes of ITER comments, including feedback to the resident, summative assessment and other more socially complex objectives.
Conclusions
Participants made inferences based on what they thought evaluators intended by their comments and seemed to share an understanding of a ‘hidden code’. Participants’ ability to ‘read between the lines’ explains how comments can be effectively used to categorise and rank‐order residents. However, it also suggests a mechanism whereby variable interpretations can arise. Our findings suggest that current assumptions about the purpose, value and effectiveness of ITER comments may be incomplete. Linguistic pragmatics and politeness theories may shed light on why such an implicit code might evolve and be maintained in clinical evaluation.
Discuss ideas arising from the article at www.mededuc.com discuss. |
---|---|
Bibliography: | National Board of Medical Examiners Stemmler Fund for Research in Education ark:/67375/WNG-C1XTVHND-3 istex:B24880A116F8E96639096C5060285BCC39237DB5 ArticleID:MEDU12637 ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
ISSN: | 0308-0110 1365-2923 |
DOI: | 10.1111/medu.12637 |