Deep learning for the identification of bruised apples by fusing 3D deep features for apple grading systems

• Published in: Mechanical systems and signal processing Vol. 145; p. 106922
• Main Authors: Hu, Zilong, Tang, Jinshan, Zhang, Ping, Jiang, Jingfeng
• Format: Journal Article
• Language: English
• Published: Berlin Elsevier Ltd 01.11.2020; Elsevier BV
• Subjects: 3D infrared sensors; Apples; Artificial neural networks; Automation; Bruised apples; Computer graphics; Deep convolutional neural network; Deep learning; Feature maps; Infrared imaging; Infrared imaging systems; Machine learning; Meshes; Model fusion; Neural networks; Performance enhancement; Performance prediction; Prediction models; Representation learning; Transfer learning; Representation learning; Meshes; Transfer learning; 3D infrared sensors; Model fusion; Deep convolutional neural network; Bruised apples
• ISSN: 0888-3270; 1096-1216
• DOI: 10.1016/j.ymssp.2020.106922

•Convolutional neural network based predictive models for the identification of bruised apples based on shape information (3D meshes) obtained by a 3D infrared imaging system are studied.•A transformation approach to represent the geometric information of 3D meshes within 2D feature maps, which aims at providing an efficient way allowing current state-of-the-art deep learning models to learn from 3D objects, is proposed.•Transfer learning is investigated in the design of the predictive models. In addition, we explore different fusion strategies to further improve the performance of the proposed models.•The identification rate for bruise apple is 97.67%. The performance distinctly exceeded the best performance of a handcraft feature engineering method proposed in our previous study. Automated bruised apple detection is an important application in the fruit industry. In this paper, we investigated convolutional neural network-based predictive models for the identification of bruised apples based on shape information (in a form of three-dimensional [3D] surface meshes) acquired from a 3D infrared imaging system. There are often irregularities on bruised apple surfaces, which can be used to differentiate those bruised apples from unbruised ones. In this study, we adopted transformation approaches so that geometric information located on 3D surface meshes could be efficiently converted to two-dimensional (2D) “image-like” feature maps. Our transformation approaches allowed current state-of-the-art deep learning models to be directly used without modifications. During algorithmic developments, three different configurations of deep convolutional neural networks were investigated in conjunction with transfer learning. We also explored different fusion strategies to further improve the performance of our predictive models. We found that the identification rate of our best predictive model was 97.67%, which markedly exceeded the best performance of a handcraft feature engineering method developed by our group. Our preliminary results show that the proposed method has great potential and further developments will lead to an automated system for apple bruise detection.