一种基于流式数据处理架构的天文元数据归档方法及系统

• Format: Patent
• Language: Chinese
• Published: 17.11.2020
• Subjects: ELECTRIC COMMUNICATION TECHNIQUE; ELECTRICITY; TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHICCOMMUNICATION

本发明请求保护一种基于流式数据处理架构的天文元数据归档方法及系统，属天文数据处理领域。首先，在数据源部署Agent。Agent实时监控数据源，一旦有新的数据产生，Agent的监控模块会立即解析新的观测数据并生成相应的元数据；其次，Agent的数据采集模块会实时采集监控模块生成的元数据，并通过网络传输到指定元数据归档平台；而后归档平台的缓存模块会实时接收数据源过来的元数据，并将元数据写入分布式缓存队列中；最后归档平台的流式数据处理组件会实时从缓存队列中读取元数据，并处理成时空维度的关联子图写入到图形数据库中。通过实时元数据归档，可减少冗余数据占据的网络带宽，减少系统占用的物理缓存成本，同时保证观测数据的实时共享。 The invention requests for protecting an astronomical metadata archiving method and system based on a streaming data processing architecture, belonging to the field of astronomical data processing. Firstly, an Agent is deployed in a data source; the Agent monitors the data source in real time; once new data is generated, the monitoring module of the Agent can instantly analyze new observation data, and generate corresponding metadata; secondly, the data acquisition module of the Agent can acquire the metadata generated by the monitoring module in real time, and transmit the metadata to an appointed metadata archiving platform through a network; then, the cache module of the archiving platform can receive