Automatic cancer nuclei segmentation on histological images: comparison study of deep learning methods

• Published in: Biotechnology and bioprocess engineering Vol. 29; no. 6; pp. 1034 - 1047
• Main Authors: Gabdullin, Maratbek T., Mukasheva, Assel, Koishiyeva, Dina, Umarov, Timur, Bissembayev, Alibek, Kim, Ki-Sub, Kang, Jeong Won
• Format: Journal Article
• Language: English
• Published: Seoul The Korean Society for Biotechnology and Bioengineering 01.12.2024; Springer Nature B.V; 한국생물공학회
• Subjects: Artificial intelligence; Biomedical engineering; Biotechnology; Cancer; Cell culture; Chemistry; Chemistry and Materials Science; color; Data analysis; Data augmentation; data collection; Data science; Deep learning; Digital imaging; Health problems; Histology; Image processing; Image segmentation; Industrial and Production Engineering; Medical imaging; neoplasm cells; Nuclei; Nuclei (cytology); Research Paper; Segmentation; 생물공학; Augmentation; Histology; Convolutional neural networks; Medical segmentation; Cancer
• ISSN: 1226-8372; 1976-3816
• DOI: 10.1007/s12257-024-00130-5

Cancer is one of the most common health problems affecting individuals worldwide. In the field of biomedical engineering, one of the main methods for cancer diagnosis is the analysis of histological images of tissue structures and cell nuclei using artificial intelligence. Here, we compared the performance of 15 deep learning methods viz: UNet, Deep-UNet, UNet-CBAM, RA-UNet, SA-Unet and Nuclei-SegNet, UNet-VGG2016, UNet-Resnet-101, TransResUNet, Inception-UNet, Att-UNet++ , FF-UNet, Att-UNet, Res-UNet and a new model, DanNucNet, in pathological nuclei segmentation on tissue slices from different organs on five open datasets: MoNuSeg, CoNSeP, CryoNuSeg, Data Science Bowl, and NuInsSeg. Before training on the data, the pixel intensity and color distribution were analyzed, and different augmentation techniques were applied. The results showed that the UNet-based model with 34.57 million Deep-UNet parameters performed the best, outperforming all models in terms of the Dice coefficient from 3.13 to 22.91%. The implementation of Deep-UNet in this context provides a valuable tool for accurate extraction of cancer cell nuclei from histological images, which in turn will contribute to further developments in cancer pathology and digital histology.