Missing Data in the American College of Surgeons National Surgical Quality Improvement Program Are Not Missing at Random: Implications and Potential Impact on Quality Assessments

• Published in: Journal of the American College of Surgeons Vol. 210; no. 2; pp. 125 - 139.e2
• Main Authors: Hamilton, Barton H., PhD, Ko, Clifford Y., MD, MS, MSHS, FACS, Richards, Karen, BA, Hall, Bruce Lee, MD, PhD, MBA, FACS
• Format: Journal Article
• Language: English
• Published: New York, NY Elsevier Inc 01.02.2010; Elsevier
• Subjects: Bias; Biological and medical sciences; Databases, Factual; General aspects; General Surgery; Health participants; Humans; Medical Errors - statistics & numerical data; Medical sciences; Models, Statistical; Preoperative Care; Public health. Hygiene; Public health. Hygiene-occupational medicine; Quality Assurance, Health Care - organization & administration; Reproducibility of Results; Research Design; Retrospective Studies; Risk Adjustment - statistics & numerical data; Serum Albumin; Societies, Medical; Surgery; United States; United States; North America; America; NSQIP; O/E ratio; relative value unit; OR; American College of Surgeons; odds ratio; VA; RVU; ACS; ratio of observed-to-expected mortality; SI; Veterans Affairs; missing at random; single imputation; National Surgical Quality Improvement Program; MAR; Medicine; Missing data; Evaluation; Improvement; Program; Treatment; American; Surgery; Quality; Quality control; Surgeon
• ISSN: 1072-7515; 1879-1190
• DOI: 10.1016/j.jamcollsurg.2009.10.021

Background Studying risk-adjusted outcomes in health care relies on statistical approaches to handling missing data. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) provides risk-adjusted assessments of surgical programs, traditionally imputing certain missing data points using a single round of multivariable imputation. Such imputation assumes that data are missing at random—without systematic bias—and does not incorporate estimation uncertainty. Alternative approaches, including using multiple imputation to incorporate uncertainty or using an indicator of missingness, can enhance robustness of evaluations. Study Design One year of de-identified data from the ACS NSQIP, representing 117 institutions and 106,113 patients, was analyzed. Using albumin variables as the missing data modeled, several imputation/adjustment models were compared, including the traditional NSQIP imputation, a new single imputation, a multiple imputation, and use of a missing indicator. Results Coefficients for albumin values changed under new single imputation and multiple imputation approaches. Multiple imputation resulted in increased standard errors, as expected. An indicator of missingness was highly explanatory, disproving the missing-at-random assumption. The effects of changes in approach differed for different outcomes, such as mortality and morbidity, and effects were greatest in smaller datasets. However, ultimate changes in patient risk assessment and institutional assessment were minimal. Conclusions Newer statistical approaches to modeling missing (albumin) values result in noticeable statistical distinctions, including improved incorporation of imputation uncertainty. In addition, the missing-at-random assumption is incorrect for albumin. Despite these findings, effects on institutional assessments are small. Although effects can be most important with smaller data-sets, the current approach to imputing missing values in the ACS NSQIP appears reasonably robust.