Analysis of rater effect in the evaluation of second language grammatical knowledge in the context of writing: Application of a generalized linear model

• Published in: Linguistic research Vol. 36; no. null; pp. 25 - 57
• Main Authors: Hyun Jung Kim, Junkyu Lee, 유현조
• Format: Journal Article
• Language: English
• Published: Seoul Kyung Hee Institute for the Study of Language and Information 01.09.2019; 언어정보연구소
• Subjects: Educational evaluation; Foreign language instruction; Grammar; Grammatical aspect; Knowledge; Linguistics; Personal essays; Second language learning; Second language writing; Second language writing instruction; Writing; Writing tests; 언어학
• ISSN: 1229-1374
• DOI: 10.17250/khisli.36..201909.002

Despite ample testing research involving the impact of rater background characteristics (e.g. raters' native language) on assessment, relatively little has been known about how raters' linguistic knowledge influences their scoring decision-making process. By associating second language acquisition research with regard to two types of linguistic knowledge (i.e. implicit and explicit knowledge), this study aims to explore the influence of raters' linguistic knowledge and other rater factors (including teaching experience) on second language writing assessment, particularly in relation to the grammatical aspect of performance. Forty-two raters' grammatical knowledge was measured with an untimed paper-and-pencil grammar test (explicit knowledge) and a timed computer-based experiment (implicit knowledge). The raters also responded to a background questionnaire. For scoring, the raters evaluated five examinees' responses on two writing tasks (a personal essay and an argumentative essay), based on a six-point analytic scoring rubric. An analysis of a cumulative link mixed model revealed that the raters' scores could be significantly related to their grammatical knowledge and teaching experience. In addition, different effects emerged across the examinees' ability levels. These findings contribute to a better understanding of the predictors with respect to rating performance and provide practical implications for rater selection and training in second language performance assessment.