Challenges and caveats of a multi-center retrospective radiomics study: an example of early treatment response assessment for NSCLC patients using FDG-PET/CT radiomics

• Published in: PloS one Vol. 14; no. 6; p. e0217536
• Main Authors: van Timmeren, Janna E, Carvalho, Sara, Leijenaar, Ralph T H, Troost, Esther G C, van Elmpt, Wouter, de Ruysscher, Dirk, Muratet, Jean-Pierre, Denis, Fabrice, Schimek-Jasch, Tanja, Nestle, Ursula, Jochems, Arthur, Woodruff, Henry C, Oberije, Cary, Lambin, Philippe
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 03.06.2019; Public Library of Science (PLoS)
• Subjects: Aged; Biology and Life Sciences; Cancer patients; Carcinoma, Non-Small-Cell Lung - diagnostic imaging; Carcinoma, Non-Small-Cell Lung - mortality; Carcinoma, Non-Small-Cell Lung - therapy; Care and treatment; CAT scans; Computer and Information Sciences; Databases, Factual; Disease-Free Survival; Female; Fluorodeoxyglucose F18 - administration & dosage; Humans; Lung cancer; Lung Neoplasms - diagnostic imaging; Lung Neoplasms - mortality; Lung Neoplasms - therapy; Male; Medical imaging equipment; Medicine and Health Sciences; Middle Aged; Models, Biological; Non-small cell lung cancer; Physical Sciences; Positron emission tomography; Positron Emission Tomography Computed Tomography; Research and Analysis Methods; Retrospective Studies; Small cell lung cancer; Survival Rate; Tomography
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0217536

Prognostic models based on individual patient characteristics can improve treatment decisions and outcome in the future. In many (radiomic) studies, small size and heterogeneity of datasets is a challenge that often limits performance and potential clinical applicability of these models. The current study is example of a retrospective multi-centric study with challenges and caveats. To highlight common issues and emphasize potential pitfalls, we aimed for an extensive analysis of these multi-center pre-treatment datasets, with an additional 18F-fluorodeoxyglucose (FDG) positron emission tomography/computed tomography (PET/CT) scan acquired during treatment. The dataset consisted of 138 stage II-IV non-small cell lung cancer (NSCLC) patients from four different cohorts acquired from three different institutes. The differences between the cohorts were compared in terms of clinical characteristics and using the so-called 'cohort differences model' approach. Moreover, the potential prognostic performances for overall survival of radiomic features extracted from CT or FDG-PET, or relative or absolute differences between the scans at the two time points, were assessed using the LASSO regression method. Furthermore, the performances of five different classifiers were evaluated for all image sets. The individual cohorts substantially differed in terms of patient characteristics. Moreover, the cohort differences model indicated statistically significant differences between the cohorts. Neither LASSO nor any of the tested classifiers resulted in a clinical relevant prognostic model that could be validated on the available datasets. The results imply that the study might have been influenced by a limited sample size, heterogeneous patient characteristics, and inconsistent imaging parameters. No prognostic performance of FDG-PET or CT based radiomics models can be reported. This study highlights the necessity of extensive evaluations of cohorts and of validation datasets, especially in retrospective multi-centric datasets.