Shadow detection and removal in apple image segmentation under natural light conditions using an ultrametric contour map

• Published in: Biosystems engineering Vol. 184; pp. 142 - 154
• Main Authors: Xu, Weiyue, Chen, Huan, Su, Qiong, Ji, Changying, Xu, Weidi, Memon, Muhammad-Sohail, Zhou, Jun
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.08.2019
• Subjects: Affinity matrix; Group pixel-wise; Image segmentation; Shadow detection; Ultrametric contour map; Image segmentation; Group pixel-wise; Shadow detection; Ultrametric contour map; Affinity matrix
• ISSN: 1537-5110; 1537-5129
• DOI: 10.1016/j.biosystemseng.2019.06.016

The image segmentation technique is a vital component of vision-based fruit harvesting robots because the accuracy of this technique greatly affects the recognition and identification by robots. However, the uncertainty and ambiguity of natural scenes in orchards make image segmentation a challenging task. Here, we developed a new algorithm to detect and remove the shadows generated under intense illumination and direct sunlight conditions. Group pixels and edge probability maps were fused in our algorithm to generate superpixel blocks with precise boundaries. We applied an affinity matrix to obtain an ultrametric contour map to detect shadows and then used a relighting method to remove the detected shadows. Additionally, the shadow detection and removal and image segmentation procedures were evaluated. Our shadow detection results showed that the root mean square error decreased from 7.9% to 6.4% when an edge probability map was applied. Using the new shadow removal algorithm, the precision, balanced accuracy, specificity, and modified segmentation accuracy were improved by 10, 11, 4.5, and 10.1%, respectively. The average segmentation processing time was 0.59 s, which meets the requirements of real-time applications (<a1 s). We conclude that the segmentation algorithm that was developed with shadow detection and removal exhibits strong robustness in detecting apples in orchards under natural light conditions. •An ultrametric contour map was introduced for image shadow detection.•The shadow detection algorithm performed well compared with previous methods.•Adding edge map, the shadow removal algorithm reduced the error by 19%.•With shadow detection and removal, the segmentation accuracy was improved by 10.1%.•The developed algorithm operated in real time using 0.59 s under natural scenes.