GeoKnowledgeFusion: A Platform for Multimodal Data Compilation from Geoscience Literature

• Published in: Remote sensing (Basel, Switzerland) Vol. 16; no. 9; p. 1484
• Main Authors: Guo, Zhixin, Wang, Chaoyang, Zhou, Jianping, Zheng, Guanjie, Wang, Xinbing, Zhou, Chenghu
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.05.2024
• Subjects: Accuracy; Algorithms; Annotations; automation; Big Data; Computational linguistics; Data collection; data extraction; data fusion; Data processing; Data science; Debris flow; Deep learning; Design; Earth science; Earth sciences research; isotopes; Language; Language processing; mass movement; Methods; multimodal data compilation; Natural language interfaces; Natural language processing; Neural networks; Samarium isotopes; scientific database; Target recognition; China
• DOI: 10.3390/rs16091484
• ISSN: 2072-4292

With the advent of big data science, the field of geoscience has undergone a paradigm shift toward data-driven scientific discovery. However, the abundance of geoscience data distributed across multiple sources poses significant challenges to researchers in terms of data compilation, which includes data collection, collation, and database construction. To streamline the data compilation process, we present GeoKnowledgeFusion, a publicly accessible platform for the fusion of text, visual, and tabular knowledge extracted from the geoscience literature. GeoKnowledgeFusion leverages a powerful network of models that provide a joint multimodal understanding of text, image, and tabular data, enabling researchers to efficiently curate and continuously update their databases. To demonstrate the practical applications of GeoKnowledgeFusion, we present two scenarios: the compilation of Sm-Nd isotope data for constructing a domain-specific database and geographic analysis, and the data extraction process for debris flow disasters. The data compilation process for these use cases encompasses various tasks, including PDF pre-processing, target element recognition, human-in-the-loop annotation, and joint multimodal knowledge understanding. The findings consistently reveal patterns that align with manually compiled data, thus affirming the credibility and dependability of our automated data processing tool. To date, GeoKnowledgeFusion has supported forty geoscience research teams within the program by processing over 40,000 documents uploaded by geoscientists.