Deep Learning-Based AI-Assisted Visual Inspection Systems for Historic Buildings and their Comparative Performance with ChatGPT-4O

• Published in: International archives of the photogrammetry, remote sensing and spatial information sciences. Vol. XLVIII-2/W8-2024; pp. 327 - 334
• Main Authors: Mishra, Mayank, Zhang, Kai, Mea, Chiara, Barazzetti, Luigi, Fassi, Francesco, Fiorillo, Fausta, Previtali, Mattia
• Format: Journal Article Conference Proceeding
• Language: English
• Published: Gottingen Copernicus GmbH 14.12.2024; Copernicus Publications
• Subjects: Agents (artificial intelligence); Annotations; Artificial intelligence; Chatbots; Decay; Deep learning; Deterioration; Historic buildings & sites; Historical buildings; Image analysis; Image degradation; Machine learning; Object recognition; Visual inspection
• DOI: 10.5194/isprs-archives-XLVIII-2-W8-2024-327-2024
• ISSN: 2194-9034; 1682-1750

Historical buildings and monuments are typically subject to degradation over time due to the passage of time and constant exposure to external agents. The use of artificial intelligence (AI) to support the work of conservation and restoration specialists in identifying surface decay is a research topic of considerable interest at present. This study presents two approaches: ChatGPT and an object detection architecture (YOLOv5). Specifically, this investigation sought to evaluate the ChatGPT’s ability to identify and describe surface degradation pathologies by exploiting its pre-trained models for image analysis. The ICOMOS-ISCS: Illustrated Glossary on Stone Deterioration Patterns (2008) was provided as a reference to guide the use of specific terminology. In the first test phase, to verify the accuracy of the ChatGPT results, benchmark images (depicting different types of damage) extracted from the UNI 11182 (2006) standard referring to the definition of degradation types were used. Only later were images from literature studies and other photographic datasets also used. In general, the results of the analysis were validated with the conclusions of professionals and with the conclusions of other AI techniques, as well as with the descriptions provided by reference manuals in the literature. In particular, the decay annotations predicted by the pre-trained object detection model were compared with those made by human experts. The capabilities and limitations of both approaches as tools for identifying deterioration pathologies are illustrated.