Terrain classification method using an NIR or RGB camera with a CNN-based fusion of vision and a reduced-order proprioception model

• Published in: Computers and electronics in agriculture Vol. 227; p. 109539
• Main Authors: Chen, Hsiao-Yu, Sang, I-Chen, Norris, William R., Soylemezoglu, Ahmet, Nottage, Dustin
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.12.2024
• Subjects: agriculture; cameras; CNN; color; electronics; Fuzzy tuner; landscapes; Near-Infrared; Outdoor terrain classification; proprioception; RGB; Sensor fusion; soil; texture; vision; Sensor fusion; Fuzzy tuner; CNN; Outdoor terrain classification; RGB; Near-Infrared
• ISSN: 0168-1699
• DOI: 10.1016/j.compag.2024.109539

•A terrain classification method integrates visual and proprioceptive signals.•The model is trained by a reduced-order convolution neural network model.•Employs a fuzzy logic tuner for adaptability in low-light conditions.•Utilizes Near-Infrared camera for improving nighttime’s overall accuracy. Terrain classification is a crucial technology for assessing terrain traversability and further assists in monitoring control and operation modes for various types of ground vehicles. The soil characteristics from the terrain can be acquired by ground vehicles adopting various sensors and physics-based dynamic feedback. Among all sensor categories, visual sensors are able to provide the most abundant information about the texture and color of terrain. However, the visual feature is limited by the amount of light available, especially in outdoor environments. This article proposes a robust terrain classification framework that fuses visual appearance and proprioceptive signals with a reduced-order model in order to solve the challenges posed by low-light conditions. The proposed one-dimensional Convolution Neural Network (CNN) model is based on raw signals from the camera’s internal IMU and the vehicle’s encoder, which can smoothly adapt to various vehicle platforms without being constrained by vehicular dimension limitations. Furthermore, the visual classification framework adapts to low-light conditions by utilizing a fuzzy tuner to adjust parameters in the Plateau Equalization (PE) process. The experimental results show that the proposed model can classify eight outdoor terrain classes with an average accuracy of 99.3 % when using a visible band camera and 99.7 % under the same experimental setup but replaced using an Near-Infrared (NIR) band camera.