Latent space manipulation for high-resolution medical image synthesis via the StyleGAN

• Published in: Zeitschrift für medizinische Physik Vol. 30; no. 4; pp. 305 - 314
• Main Authors: Fetty, Lukas, Bylund, Mikael, Kuess, Peter, Heilemann, Gerd, Nyholm, Tufve, Georg, Dietmar, Löfstedt, Tommy
• Format: Journal Article
• Language: English
• Published: Germany Elsevier GmbH 01.11.2020
• Subjects: Algorithms; Deep Learning; Female; Humans; Image Processing, Computer-Assisted - methods; Image synthesis; Latent space; Magnetic Resonance Imaging - methods; Male; Pelvic Neoplasms - diagnostic imaging; Pelvic Neoplasms - radiotherapy; StyleGAN; StyleGAN, Image synthesis, Latent space; Tomography, X-Ray Computed; StyleGAN, Image synthesis, Latent space
• ISSN: 0939-3889; 1876-4436; 1876-4436
• DOI: 10.1016/j.zemedi.2020.05.001

This paper explores the potential of the StyleGAN model as an high-resolution image generator for synthetic medical images. The possibility to generate sample patient images of different modalities can be helpful for training deep learning algorithms as e.g. a data augmentation technique. The StyleGAN model was trained on Computed Tomography (CT) and T2- weighted Magnetic Resonance (MR) images from 100 patients with pelvic malignancies. The resulting model was investigated with regards to three features: Image Modality, Sex, and Longitudinal Slice Position. Further, the style transfer feature of the StyleGAN was used to move images between the modalities. The root-mean-squard error (RMSE) and the Mean Absolute Error (MAE) were used to quantify errors for MR and CT, respectively. We demonstrate how these features can be transformed by manipulating the latent style vectors, and attempt to quantify how the errors change as we move through the latent style space. The best results were achieved by using the style transfer feature of the StyleGAN (58.7 HU MAE for MR to CT and 0.339 RMSE for CT to MR). Slices below and above an initial central slice can be predicted with an error below 75 HU MAE and 0.3 RMSE within 4cm for CT and MR, respectively. The StyleGAN is a promising model to use for generating synthetic medical images for MR and CT modalities as well as for 3D volumes.