NaroNet: Discovery of tumor microenvironment elements from highly multiplexed images

• Published in: Medical image analysis Vol. 78; no. C; p. 102384
• Main Authors: Jiménez-Sánchez, Daniel, Ariz, Mikel, Chang, Hang, Matias-Guiu, Xavier, de Andrea, Carlos E., Ortiz-de-Solórzano, Carlos
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier B.V 01.05.2022; Elsevier BV; Elsevier
• Subjects: Breast cancer; Breast Neoplasms - diagnostic imaging; Cancer; Cellular neighborhoods; Computer applications; Computer Science; Cytometry; Deep learning; Endometrial cancer; Endometrium; Engineering; Female; Fibroblasts; Humans; Image processing; Imaging mass cytometry; Immune system; Interpretable machine learning; Medical imaging; Multiplex imaging; Multiplexing; Patients; Phenotypes; Predictions; Prognosis; Radiology, Nuclear Medicine & Medical Imaging; Self supervised learning; Spatial biology; Tissues; Tumor cells; Tumor Microenvironment; Tumors; Weakly supervised learning; Deep learning; Self supervised learning; 92C55; Tumor microenvironment; Spatial biology; Weakly supervised learning; Interpretable machine learning; Multiplex imaging; Imaging mass cytometry; Cellular neighborhoods
• ISSN: 1361-8415; 1361-8423
• DOI: 10.1016/j.media.2022.102384

•First weakly-supervised DL framework that learns the tumor microenvironment from highly multiplexed images.•Learning the tumor microenvironment in three levels of spatial complexity.•Blindly associates tumor microenvironment elements with clinical information.•Demonstrated accuracy and interpretability on both real and synthetic datasets. [Display omitted] Understanding the spatial interactions between the elements of the tumor microenvironment -i.e. tumor cells. fibroblasts, immune cells- and how these interactions relate to the diagnosis or prognosis of a tumor is one of the goals of computational pathology. We present NaroNet, a deep learning framework that models the multi-scale tumor microenvironment from multiplex-stained cancer tissue images and provides patient-level interpretable predictions using a seamless end-to-end learning pipeline. Trained only with multiplex-stained tissue images and their corresponding patient-level clinical labels, NaroNet unsupervisedly learns which cell phenotypes, cell neighborhoods, and neighborhood interactions have the highest influence to predict the correct label. To this end, NaroNet incorporates several novel and state-of-the-art deep learning techniques, such as patch-level contrastive learning, multi-level graph embeddings, a novel max-sum pooling operation, or a metric that quantifies the relevance that each microenvironment element has in the individual predictions. We validate NaroNet using synthetic data simulating multiplex-immunostained images where a patient label is artificially associated to the -adjustable- probabilistic incidence of different microenvironment elements. We then apply our model to two sets of images of human cancer tissues: 336 seven-color multiplex-immunostained images from 12 high-grade endometrial cancer patients; and 382 35-plex mass cytometry images from 215 breast cancer patients. In both synthetic and real datasets, NaroNet provides outstanding predictions of relevant clinical information while associating those predictions to the presence of specific microenvironment elements.