Integration of ultrasound and mammogram for multimodal classification of breast cancer using hybrid residual neural network and machine learning

• Published in: Image and vision computing Vol. 145; p. 104987
• Main Authors: Atrey, Kushangi, Singh, Bikesh Kumar, Bodhey, Narendra Kuber
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.05.2024
• Subjects: Breast cancer; Feature fusion; Machine learning; Multimodal classification; Transfer learning; Multimodal classification; Breast cancer; Feature fusion; Transfer learning; Machine learning
• ISSN: 0262-8856; 1872-8138
• DOI: 10.1016/j.imavis.2024.104987

Breast cancer (BC) is one of the topmost causes of mortality in women all over the world. Early detection and classification of the tumor allow proper treatment of patients and chances of survival. In this article, we propose a hybrid residual neural network (ResNet) and machine learning framework and integrate the features of both mammography (MG) and ultrasound (US) images to perform the multimodal classification of BC images as benign or malignant. The features are extracted automatically from the input images of each modality using the residual neural network from the average pooling layer. Next, the feature level fusion is carried out to obtain a feature vector by combining features of MG & US. Finally, the multimodal classification is performed using the support vector machine (SVM) as a classifier. Experiments are performed on a real-time dataset collected from patients who have undergone both MG and US examinations. The classification accuracy obtained for the multimodal approach with SVM is 99.22%, which is higher than unimodal systems. Results show that the proposed multimodal approach performs better in classifying breast tumors than unimodal mammogram and ultrasound systems. [Display omitted] •A multimodal classification using combined ultrasound and mammogram is presented.•To improve the diversity of the dataset, data augmentation is applied.•Hybrid ResNet-SVM architecture is proposed.•The ResNet extracts discriminant features automatically from both modalities.•Proposed multimodal system outperforms unimodal mammogram and ultrasound systems.