Diffusion-based data augmentation and hierarchical CLIP for real estate image annotation

• Published in: Pattern analysis and applications : PAA Vol. 28; no. 2
• Main Authors: Deng, Haojin, Zhang, Wandong, Yang, Yimin, Nejad, Eman
• Format: Journal Article
• Language: English
• Published: London Springer London 01.06.2025; Springer Nature B.V
• Subjects: Algorithms; Computer Science; Data augmentation; Houses; Image annotation; Image classification; Original Article; Pattern Recognition; Quality assessment; Real estate; Search engines; Deep learning; Contrastive language-image pretraining; Diffusion model; Real estate image
• ISSN: 1433-7541; 1433-755X
• DOI: 10.1007/s10044-025-01465-2

The critical role of real estate search engines in the economic domain underscores the necessity for a robust methodology in annotating the luxury level of room images. Contemporary challenges include inadequacies in room quality assessment, underutilization of deep network capacities, and insufficient annotated house images. This paper presents an innovative real estate image annotation model employing a multi-stage approach that integrates diffusion models with contrastive language-image pretraining (CLIP) networks. Initially, the diffusion network serves as a data augmentation technique, generating supplementary real estate images for training. Next, a hierarchical CLIP model categorizes images into different room types. Subsequently, multiple CLIP models assess the condition of each room, annotating them as either contemporary or standard. The final and most significant stage involves transferring knowledge from the larger CLIP model to a smaller, more efficient model using soft labeling. This approach enhances inferencing speed while maintaining high performance. Experimental results on a newly gathered real estate image dataset demonstrate the superior performance of the proposed approach compared to existing house image classification algorithms. Our demonstration code and checkpoints are available at https://huggingface.co/strollingorange/roomLuxuryAnnotater .