Judgment analysis: a method for quantitative evaluation of trainee surgeons’ judgments of surgical risk

• Published in: The American journal of surgery Vol. 195; no. 2; pp. 183 - 188
• Main Authors: Jacklin, Ros, M.B.B.S., M.A., M.R.C.S, Sevdalis, Nick, B.Sc., M.Sc., Ph.D, Harries, Clare, B.Sc., Ph.D, Darzi, Ara, M.B.B.S., M.D, Vincent, Charles, B.A., M.Phil., Ph.D
• Format: Journal Article
• Language: English
• Published: New York, NY Elsevier Inc 01.02.2008; Elsevier; Elsevier Limited
• Subjects: Abdomen; Abdominal surgery; Adult; Biological and medical sciences; Cholecystectomy - education; Cholecystectomy - methods; Cholecystectomy, Laparoscopic - education; Cholecystectomy, Laparoscopic - methods; Clinical Competence; Comorbidity; Decision Making; Education, Medical, Graduate; Evaluation Studies as Topic; Feedback; Female; Gallbladder; Gallbladder diseases; Gallstones; General aspects; Health participants; Human performance; Humans; Judgment; Judgment analysis; Male; Medical Errors - prevention & control; Medical sciences; Models, Theoretical; Mortality; Patients; Program Evaluation; Public health. Hygiene; Public health. Hygiene-occupational medicine; Reproducibility of Results; Risk; Risk Assessment; Sensitivity and Specificity; Studies; Surgery; Surgical outcomes; United Kingdom > UK; Judgment analysis; Risk; Decision making; Surgery; Judgment; Evaluation; Epidemiology; Surgeon; Medicine; Analysis method; Treatment; Risk factor; Quantitative analysis
• ISSN: 0002-9610; 1879-1883
• DOI: 10.1016/j.amjsurg.2007.09.031

Abstract Background Surgical judgment and decision making require valid methods of assessment. The aim of this study was to evaluate the use of judgment analysis as a technique for quantitative evaluation of surgeons’ risk estimates. Methods Thirty trainee surgeons’ estimates of conversion risk in laparoscopic cholecystectomy were investigated using judgment analysis. Hypothetical cases were created, differing in relevant risk factors. Twenty repeat cases were incorporated to test for reliability. Surgeons’ estimates were compared with an outcome-derived gold standard from the published literature. Results The mean reliability was .77 (range, .47–.98), and regression models indicating the weighting of variables had a mean adjusted R2 value of .53 (range, .12–.76). Variables were subject to wide variation in weighting. The mean correlation to the gold standard model was .48 (range, .08–.72). Conclusions Judgment analysis allows detailed quantitative evaluation of the consistency of surgeons’ risk estimates and the influence of different variables on them. Comparison with a gold standard model enables accuracy to be measured.