Method and device for data deduplication based on K-V storage engine

• Main Authors: ZHANG YAO, NI SHILONG, SU JIANGWEN
• Format: Patent
• Language: Chinese; English
• Published: 05.04.2017
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

The invention discloses a method and a device for data deduplication based on a K-V storage engine. The method comprises the following steps of splitting a file into data blocks with the fixed length; carrying out Hash computation on each data block to obtain a data block abstract array; and comparing abstracts in the array with a data block abstract stored in the K-V storage engine one by one, if the same abstract exists, not carrying out storage, otherwise storing into the K-V storage engine with the data block abstract as a Key and a to-be-stored data block as a Value. 种基于K-V存储引擎的数据去重方法及装置，其中方法包括如下步骤，将文件分割成定长的数据块，对各数据块进行哈希计算，得到数据块摘要数组，将数组中摘要逐个与已存储在K-V存储引擎中的数据块摘要进行比对，若存在相同的摘要，则不进行存储，否则以数据块摘要为Key，待存储数据块为Value存入K-V存储引擎。