Convolution-Transformer for Image Feature Extraction

• Published in: Computer modeling in engineering & sciences Vol. 141; no. 1; pp. 87 - 106
• Main Authors: Yin, Lirong, Wang, Lei, Lu, Siyu, Wang, Ruiyang, Yang, Youshuai, Yang, Bo, Liu, Shan, AlSanad, Ahmed, AlQahtani, Salman A., Yin, Zhengtong, Li, Xiaolu, Chen, Xiaobing, Zheng, Wenfeng
• Format: Journal Article
• Language: English
• Published: Henderson Tech Science Press 2024
• Subjects: Accuracy; Artificial neural networks; Bias; Convergence; Feature extraction; Image classification; Invariance; Neural networks; Scale invariance; Stability
• DOI: 10.32604/cmes.2024.051083
• ISSN: 1526-1506; 1526-1492

This study addresses the limitations of Transformer models in image feature extraction, particularly their lack of inductive bias for visual structures. Compared to Convolutional Neural Networks (CNNs), the Transformers are more sensitive to different hyperparameters of optimizers, which leads to a lack of stability and slow convergence. To tackle these challenges, we propose the Convolution-based Efficient Transformer Image Feature Extraction Network (CEFormer) as an enhancement of the Transformer architecture. Our model incorporates E-Attention, depthwise separable convolution, and dilated convolution to introduce crucial inductive biases, such as translation invariance, locality, and scale invariance, into the Transformer framework. Additionally, we implement a lightweight convolution module to process the input images, resulting in faster convergence and improved stability. This results in an efficient convolution combined Transformer image feature extraction network. Experimental results on the ImageNet1k Top-1 dataset demonstrate that the proposed network achieves better accuracy while maintaining high computational speed. It achieves up to 85.0% accuracy across various model sizes on image classification, outperforming various baseline models. When integrated into the Mask Region-Convolutional Neural Network (R-CNN) framework as a backbone network, CEFormer outperforms other models and achieves the highest mean Average Precision (mAP) scores. This research presents a significant advancement in Transformer-based image feature extraction, balancing performance and computational efficiency.