Building Type Classification Using CNN-Transformer Cross-Encoder Adaptive Learning From Very High Resolution Satellite Images

• Published in: IEEE journal of selected topics in applied earth observations and remote sensing Vol. 18; pp. 976 - 994
• Main Authors: Zhang, Shaofeng, Li, Mengmeng, Zhao, Wufan, Wang, Xiaoqin, Wu, Qunyong
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.01.2025; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Architecture; Artificial neural networks; Attention; Boundaries; Building type classification; Buildings; Classification; CNN-transformer networks; Coders; cross-encoder; Datasets; Earth; Feature extraction; feature interaction; High resolution; Image enhancement; Image resolution; Machine learning; Modules; Neural networks; Optimization; Remote sensing; Satellite imagery; Semantics; Transformers; Urban environments; very high resolution remote sensing; Visual discrimination learning; Visual perception; Visualization
• DOI: 10.1109/JSTARS.2024.3501678
• ISSN: 1939-1404

Building type information indicates the functional properties of buildings and plays a crucial role in smart city development and urban socioeconomic activities. Existing methods for classifying building types often face challenges in accurately distinguishing buildings between types while maintaining well-delineated boundaries, especially in complex urban environments. This study introduces a novel framework, i.e., CNN-Transformer cross-attention feature fusion network (CTCFNet), for building type classification from very high resolution remote sensing images. CTCFNet integrates convolutional neural networks (CNNs) and Transformers using an interactive cross-encoder fusion module that enhances semantic feature learning and improves classification accuracy in complex scenarios. We develop an adaptive collaboration optimization module that applies human visual attention mechanisms to enhance the feature representation of building types and boundaries simultaneously. To address the scarcity of datasets in building type classification, we create two new datasets, i.e., the urban building type (UBT) dataset and the town building type (TBT) dataset, for model evaluation. Extensive experiments on these datasets demonstrate that CTCFNet outperforms popular CNNs, Transformers, and dual-encoder methods in identifying building types across various regions, achieving the highest mean intersection over union of 78.20% and 77.11%, F1 scores of 86.83% and 88.22%, and overall accuracy of 95.07% and 95.73% on the UBT and TBT datasets, respectively. We conclude that CTCFNet effectively addresses the challenges of high interclass similarity and intraclass inconsistency in complex scenes, yielding results with well-delineated building boundaries and accurate building types.