Tactile texture recognition of multi-modal bionic finger based on multi-modal CBAM-CNN interpretable method

• Published in: Displays Vol. 83; p. 102732
• Main Authors: Ma, Feihong, Li, Yuliang, Chen, Meng
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.07.2024
• Subjects: Data fusion; Feature fusion; Multi-modal bionic finger tactile sensor; multi-modal CAM feature visualize; multi-modal CBAM-CNN; Wavelet transform; Data fusion; Feature fusion; multi-modal CAM feature visualize; Wavelet transform; multi-modal CBAM-CNN; Multi-modal bionic finger tactile sensor
• ISSN: 0141-9382; 1872-7387
• DOI: 10.1016/j.displa.2024.102732

•A multi-modal bionic finger tactile sensor and its experimental platform are proposed, which contains 4 exciting electrodes, 14 sensing electrodes, 1 pressure sensor and 1 temperature sensor.•The correlation function weighted fusion method is used for 14-channel electrode signals. The wavelet transform is used for 14-channel electrode signals, pressure signals and temperature signals.•The multi-modal CBAM-CNN network is established to extract the multi-modal tactile features after the wavelet transform and the multi-modal tactile features are fused in feature level.•The improved multi-modal CAM method based on multi-modal CBAM-CNN model is proposed to understand the classification decision of each modal intuitively. To solve the problem of lacking surface information while using a single-modal sensor to scan fine textures, a multi-modal bionic finger tactile sensor is used to perceive the information contained in textures, and a multi-modal tactile texture recognition method is proposed to address the issue of poor interpretability for tactile texture recognition models. First, the experimental platform of multi-modal bionic finger tactile texture information acquisition system is built, which can simultaneously collect 14 channels of tactile electrode signals, one channel of pressure signals and one channel of temperature signals. Second, the multi-modal tactile signals can be fused at the data level after signal processing, and then the wavelet transform method is carried out on the basis of data fusion. Third, a multi-modal CBAM-CNN network (Convolutional Block Attention Module, Convolutional Neural Network) is established to extract the multi-modal tactile features after the wavelet transform and the multi-modal tactile features are fused at feature level. To verify the performance of multi-modal CBAM-CNN model, 27(3*9) kinds of millimeter-level fine textures of 3 datasets are tested and the recognition accuracy of each dataset is 98.47%, 97.63%, 99.16%. And it can also be seen that the 14-channel modal data contributes the most to the performance of multi-modal CBAM-CNN model for various textures. The results of ablation experiments prove that the identification accuracy of the fused multi-modal CBAM-CNN is higher than the single-modal CBAM-CNN no matter in time domain or time–frequency domain. Moreover, the accuracy of the weighted fusion method based on correlation function is higher than horizontal fusion method in 14 channels of tactile electrode signals modal. Finally, to interpret the classification decision of the multi-modal CBAM-CNN network in each modal intuitively, the improved multi-modal CAM (Class Activation Map) algorithm is adopted to visualize the interest regions of the samples in detail, and the activated areas in the map make the identification result of CBAM-CNN in each modal more transparent and interpretable.