Hive and Spark-based remote sensing data service offline batch processing system and method

• Main Authors: ZANG WENQIAN, QIAO BAOJUN, ZHANG HONGKUO, ZHANG LEI, ZUO XIANYU, LIN YINGHAO
• Format: Patent
• Language: Chinese; English
• Published: 14.12.2021
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

The invention provides a Hive and Spark-based remote sensing data service offline batch processing system and a data processing method thereof. The system comprises: an acquisition module, a storage module, a calculation module, a data service interface, and a data application module, wherein the calculation module comprises iterative memory calculation Spark and uniform resource scheduling management Yarn; the system further comprises a Hive data warehouse, wherein the Hive data warehouse comprises: an original data ODS layer, a dimension data DIM layer, a detail data EDW layer, a summary data CDM layer, and an application data APP layer. According to the system and the method, Hive and Spark computing engines are used for performing offline processing on the remote sensing data service, so that data support is provided for meeting diversified requirements of users and realizing an active service mode. 本发明提供一种基于Hive和Spark的遥感数据服务离线批处理系统及其数据处理方法。该系统包括采集模块、存储模块、计算模块、数据服务接口和数据应用模块，所述计算模块包括迭代式内存计算Spark和统一资源调度管理Ya