Extending scene-to-patch models: Multi-resolution multiple instance learning for Earth observation

• Published in: Environmental Data Science Vol. 2
• Main Authors: Early, Joseph, Deweese, Ying-Jung Chen, Evers, Christine, Ramchurn, Sarvapali
• Format: Journal Article
• Language: English
• Published: Cambridge University Press 2023
• Subjects: Earth observation; multi-resolution; multiple instance learning; remote sensing; scene-to-patch models
• DOI: 10.1017/eds.2023.30
• ISSN: 2634-4602

Abstract Land cover classification (LCC) and natural disaster response (NDR) are important issues in climate change mitigation and adaptation. Existing approaches that use machine learning with Earth observation (EO) imaging data for LCC and NDR often rely on fully annotated and segmented datasets. Creating these datasets requires a large amount of effort, and a lack of suitable datasets has become an obstacle in scaling the use of machine learning for EO. In this study, we extend our prior work on Scene-to-Patch models: an alternative machine learning approach for EO that utilizes Multiple Instance Learning (MIL). As our approach only requires high-level scene labels, it enables much faster development of new datasets while still providing segmentation through patch-level predictions, ultimately increasing the accessibility of using machine learning for EO. We propose new multi-resolution MIL architectures that outperform single-resolution MIL models and non-MIL baselines on the DeepGlobe LCC and FloodNet NDR datasets. In addition, we conduct a thorough analysis of model performance and interpretability.