Interrater and Intrarater Reliability of the Colloid Cyst Risk Score

• Published in: Neurosurgery Vol. 86; no. 1; pp. E47 - E53
• Main Authors: Alford, Elizabeth N, Rotman, Lauren E, Lepard, Jacob R, Agee, Bonita S, Markert, James M
• Format: Journal Article
• Language: English
• Published: United States Oxford University Press 01.01.2020; Copyright by the Congress of Neurological Surgeons; Wolters Kluwer Health, Inc
• Subjects: Brain diseases; Cerebrovascular disease; Complications and side effects; Cysts; Diagnosis; Evaluation; Headaches; Health risk assessment; Hydrocephalus; Medical tests; Neurosurgery; Risk factors; United States; Colloid cyst; Interrater reliability; Hydrocephalus; Intrarater reliability
• ISSN: 0148-396X; 1524-4040
• DOI: 10.1093/neuros/nyz399

Abstract BACKGROUND The Colloid Cyst Risk Score (CCRS) was developed to identify symptomatic patients and stratify risk of hydrocephalus among patients with colloid cysts. Its components consider patient age, cyst diameter, presence/absence of headache, fluid-attenuated inversion recovery (FLAIR) hyperintensity, and location within the third ventricle. OBJECTIVE To independently evaluate the inter- and intrarater reliability of the CCRS. METHODS Patients with a colloid cyst were identified from billing records and radiology archives. Three independent raters reviewed electronic medical records to determine age, presence/absence of headache, cyst diameter (mm), FLAIR hyperintensity, and risk zone location. Raters made 53 observations, including 5 repeat observations. Fleiss’ generalized kappa (κ) was calculated for all of the nominal criteria, whereas Kendall's coefficient of concordance (W) and the intraclass correlation coefficient (ICC) were calculated for the overall score. RESULTS Total CCRS score demonstrated extremely strong agreement (W = 0.83) using Kendall's W coefficient and good agreement (ICC = 0.74) using the ICC (P < .001). For interrater reliability of individual criteria, age (κ = 1.00) and FLAIR hyperintensity (κ = 0.89) demonstrated near perfect agreement. Axial diameter (κ = 0.63) demonstrated substantial agreement, whereas agreement was moderate for risk zone (κ = 0.51) and fair for headache (κ = 0.26). Intrarater reliability for total CCRS score was extremely strong using Kendall's W, good to excellent using ICC, and fair to substantial using weighted kappa. CONCLUSION The CCRS has good inter- and intrarater reliability when tested in an independent sample of patients, though strength of agreement varies among individual criteria. The validity of the CCRS requires independent evaluation. Graphical Abstract Graphical Abstract