Automatic classification of mangosteens and ripe status in images using deep learning based approaches

• Published in: Multimedia tools and applications Vol. 83; no. 16; pp. 48275 - 48290
• Main Authors: Kusakunniran, Worapan, Imaromkul, Thanandon, Aukkapinyo, Kittinun, Thongkanchorn, Kittikhun, Somsong, Pimpinan
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.05.2024; Springer Nature B.V
• Subjects: Artificial neural networks; Classification; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Datasets; Deep learning; Image classification; Machine learning; Multimedia Information Systems; Special Purpose and Application-Based Systems; Deep learning; Multi-class classification; Mangosteens classification; Ripe status; Binary classification
• ISSN: 1573-7721; 1380-7501; 1573-7721
• DOI: 10.1007/s11042-023-17505-0

After the mangosteen-harvest, it is necessary to process a grading assessment which contains a key step of classifying a ripe status. This paper aims to develop an automatic solution to replace a manual process by human experts for classifying mangosteen and their ripe status in images. The classification solutions are developed based on deep learning techniques. These classification models are constructed by attempting on four architectures (i.e. DenseNet, EfficientNet, ResNet, and VGG) of convolutional neural networks (CNN). The models are trained using well-known and new prepared datasets. Two training strategies of multi-class and binary classifications are attempted in our experiments for distinguishing mangosteen from other fruits. It is reported that the multi-class classification performs better than the binary classification, with the precision, recall, and f1-score of 100%. In addition, a gradient-weighted class activation mapping (Grad-CAM) is used to demonstrate the reliability of the trained models. The proposed solution based on EfficientNetB0 performs the best for classification of mangosteens and their ripe statuses with the average accuracies of 100% and 98% respectively. The multi-class CNN-based classification is developed for solving a real-world problem of the ripe status classification. Alternative CNN architectures are attempted for finding the best-fit solution on a publicly available dataset and a self-collected dataset from a web scraping. The computed heatmaps show that it is not necessary to perform the mangosteen segmentation, the classification task could be performed directly where background and irrelevant parts of images are not/or less used.