Microclimate Vision: Multimodal prediction of climatic parameters using street-level and satellite imagery

• Published in: Sustainable cities and society Vol. 114; p. 105733
• Main Authors: Fujiwara, Kunihiko, Khomiakov, Maxim, Yap, Winston, Ignatius, Marcel, Biljecki, Filip
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.11.2024
• Subjects: Climate change adaptation; Computer vision; Heat risk management; Solar radiation; Street-view imagery; Urban morphology; Urban thermal environment; Computer vision; Urban thermal environment; Street-view imagery; Heat risk management; Solar radiation; Climate change adaptation; Urban morphology
• ISSN: 2210-6707
• DOI: 10.1016/j.scs.2024.105733

High-resolution microclimate data is essential for capturing spatio-temporal heterogeneity of urban climate and heat health management. However, previous studies have relied on dense measurements that require significant costs for equipment, or on physical simulations demanding intensive computational loads. As a potential alternative to these methods, we propose a multimodal deep learning model to predict microclimate at a high spatial and temporal resolution based on street-level and satellite imagery. This model consists of LSTM and ResNet-18 architectures, and predicts air temperature (Tair), relative humidity (RH), wind speed (ν), and global horizontal irradiance (GHI). For our study area situated at a university campus in Singapore, we collected microclimate data, street-level and satellite imagery. We conducted extensive experiments with our collected dataset to showcase our model’s predictive capabilities and its practical use in generating high-resolution microclimate maps. Our model reported RMSE at 0.95 °C for Tair, 2.57% for RH, 0.31 m/s for ν, and 225 W/m2 for GHI. Furthermore, we observed a contribution of imagery inputs to higher accuracy by comparing models with and without such inputs. We identified hot spots at a high spatio-temporal resolution, indicating its application for issuing real-time heat alerts. Our models are released openly at the microclimate-vision GitHub repository (https://github.com/kunifujiwara/microclimate-vision). [Display omitted] •Prediction of microclimate parameters using street-level and satellite imagery.•Multimodal prediction model combining LSTM and ResNet-18 architectures.•Collecting and using microclimate data, street-level imagery, and satellite imagery.•High accuracy: RMSE at 0.95 °C (Tair), 2.57% (RH), 0.31 m/s (ν), 225 W/m2 (GHI).•Street-level and satellite imagery contribute to improving accuracy.