Data processing method and device
The invention discloses a data processing method and device. The data processing method comprises the steps: cleaning target data from the data, wherein the target data comprise fields and values of data in the form of messages; performing de-duplication processing on the plurality of pieces of targ...
Saved in:
| Main Authors | , , |
|---|---|
| Format | Patent |
| Language | Chinese English |
| Published |
06.09.2019
|
| Subjects | |
| Online Access | Get full text |
Cover
Loading…
| Abstract | The invention discloses a data processing method and device. The data processing method comprises the steps: cleaning target data from the data, wherein the target data comprise fields and values of data in the form of messages; performing de-duplication processing on the plurality of pieces of target data; and storing the multiple pieces of duplicate-removed target data in storage files of multiple tables in a landing mode at the same time, and guaranteeing that the number of the storage files is controllable and the size is balanced. The data processing method can solve the technical problems that in the prior art, only the data of a single table can be landed, and efficiency is low.
本发明公开了一种数据处理方法及装置。其中，该方法包括：从数据中清洗出目标数据，其中，目标数据包括报文形式的数据的字段和值；对多个目标数据进行去重处理；将去重后的多个目标数据，同时落地存储到多个表的存储文件中，且保证存储文件数量可控、大小均衡。本发明解决了相关技术中仅能对单个表的数据落地的方式，效率低的技术问题。 |
|---|---|
| AbstractList | The invention discloses a data processing method and device. The data processing method comprises the steps: cleaning target data from the data, wherein the target data comprise fields and values of data in the form of messages; performing de-duplication processing on the plurality of pieces of target data; and storing the multiple pieces of duplicate-removed target data in storage files of multiple tables in a landing mode at the same time, and guaranteeing that the number of the storage files is controllable and the size is balanced. The data processing method can solve the technical problems that in the prior art, only the data of a single table can be landed, and efficiency is low.
本发明公开了一种数据处理方法及装置。其中，该方法包括：从数据中清洗出目标数据，其中，目标数据包括报文形式的数据的字段和值；对多个目标数据进行去重处理；将去重后的多个目标数据，同时落地存储到多个表的存储文件中，且保证存储文件数量可控、大小均衡。本发明解决了相关技术中仅能对单个表的数据落地的方式，效率低的技术问题。 |
| Author | XING QINHUA YIN SUNA ZHENG BAIYUN |
| Author_xml | – fullname: YIN SUNA – fullname: XING QINHUA – fullname: ZHENG BAIYUN |
| BookMark | eNrjYmDJy89L5WRQdEksSVQoKMpPTi0uzsxLV8hNLcnIT1FIzEtRSEkty0xO5WFgTUvMKU7lhdLcDIpuriHOHrqpBfnxqcUFicmpeakl8c5-hoYGRgaWZqZmjsbEqAEARSwnHQ |
| ContentType | Patent |
| DBID | EVB |
| DatabaseName | esp@cenet |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: EVB name: esp@cenet url: http://worldwide.espacenet.com/singleLineSearch?locale=en_EP sourceTypes: Open Access Repository |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Medicine Chemistry Sciences Physics |
| DocumentTitleAlternate | 数据处理方法及装置 |
| ExternalDocumentID | CN110209656A |
| GroupedDBID | EVB |
| ID | FETCH-epo_espacenet_CN110209656A3 |
| IEDL.DBID | EVB |
| IngestDate | Fri Jul 19 15:06:05 EDT 2024 |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | false |
| Language | Chinese English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-epo_espacenet_CN110209656A3 |
| Notes | Application Number: CN201910345365 |
| OpenAccessLink | https://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20190906&DB=EPODOC&CC=CN&NR=110209656A |
| ParticipantIDs | epo_espacenet_CN110209656A |
| PublicationCentury | 2000 |
| PublicationDate | 20190906 |
| PublicationDateYYYYMMDD | 2019-09-06 |
| PublicationDate_xml | – month: 09 year: 2019 text: 20190906 day: 06 |
| PublicationDecade | 2010 |
| PublicationYear | 2019 |
| RelatedCompanies | BEIJING HUJIN XINRONG TECHNOLOGY CO., LTD |
| RelatedCompanies_xml | – name: BEIJING HUJIN XINRONG TECHNOLOGY CO., LTD |
| Score | 3.346682 |
| Snippet | The invention discloses a data processing method and device. The data processing method comprises the steps: cleaning target data from the data, wherein the... |
| SourceID | epo |
| SourceType | Open Access Repository |
| SubjectTerms | CALCULATING COMPUTING COUNTING ELECTRIC DIGITAL DATA PROCESSING PHYSICS |
| Title | Data processing method and device |
| URI | https://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20190906&DB=EPODOC&locale=&CC=CN&NR=110209656A |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV3dS8MwED_m_HzTquj8oAPpW7Fru6x7KOLSliGsGzJlbyNpE6YPs9iK4F_vJeucL_oWLpCPg19-d8ndBeDGY25XIu8ikKRn-9LrYIsLmxE3I1xKHjgqUXiUkuGT_zDrzhrwus6F0XVCP3VxRERUhniv9HldbC6xIh1bWd7yFxS93SXTMLJq7xjZre8QKxqE8WQcjalFaUhTK30MkeVcVeiE3G_BNprRPYWG-HmgslKK35SSHMLOBEdbVkfQ-FoYsE_XP68ZsDeqH7wN2NURmlmJwhqF5TG0I1Yxs1iF-CP1mKtvoE22zM1cKOifQDuJp3Ro46zzny3OabpZoHcKTXT9xRmYrkOkn6EJEnSkL_oBZ75w0WLKmbK0st45tP4ep_Vf5wUcKHXpaClyCc3q_UNcIb1W_Frr5Rv1rnz4 |
| link.rule.ids | 230,309,783,888,25576,76876 |
| linkProvider | European Patent Office |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LT4NAEJ7U-qg3RRutL5oYbkQKdEsPxNgFglpoY9D01iyUjXqoxGJM_PXObqn1orfNbLKPSb79ZnZnZgEuLWZ2OfIuAolbus2tDrbSXGfEzEjKeeoYIlE4ikn4aN9NupMavK5yYWSd0E9ZHBERlSHeS3leF-tLLE_GVi6u0hcUvV0HietplXeM7NY3iOYNXH888kZUo9SlsRY_uMhypih0Qm42YBNN7J5Ag_80EFkpxW9KCfZga4yjzct9qH09K9Cgq5_XFNiJqgdvBbZlhGa2QGGFwsUBtD1WMrVYhvgj9ajLb6BVNp-ps1xA_xDagZ_QUMdZpz9bnNJ4vUCrCXV0_fMjUE2DcDtDE8TpcDvvOymzcxMtphkTllbWO4bW3-O0_uu8gEaYRMPp8Da-P4FdoToZOUVOoV6-f-RnSLVlei519A38in_r |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Apatent&rft.title=Data+processing+method+and+device&rft.inventor=YIN+SUNA&rft.inventor=XING+QINHUA&rft.inventor=ZHENG+BAIYUN&rft.date=2019-09-06&rft.externalDBID=A&rft.externalDocID=CN110209656A |