Data processing method and device

• Main Authors: YIN SUNA, XING QINHUA, ZHENG BAIYUN
• Format: Patent
• Language: Chinese; English
• Published: 06.09.2019
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

The invention discloses a data processing method and device. The data processing method comprises the steps: cleaning target data from the data, wherein the target data comprise fields and values of data in the form of messages; performing de-duplication processing on the plurality of pieces of target data; and storing the multiple pieces of duplicate-removed target data in storage files of multiple tables in a landing mode at the same time, and guaranteeing that the number of the storage files is controllable and the size is balanced. The data processing method can solve the technical problems that in the prior art, only the data of a single table can be landed, and efficiency is low. 本发明公开了一种数据处理方法及装置。其中，该方法包括：从数据中清洗出目标数据，其中，目标数据包括报文形式的数据的字段和值；对多个目标数据进行去重处理；将去重后的多个目标数据，同时落地存储到多个表的存储文件中，且保证存储文件数量可控、大小均衡。本发明解决了相关技术中仅能对单个表的数据落地的方式，效率低的技术问题。