Erasure-Coding-Based Storage and Recovery for Distributed Exascale Storage Systems

Various techniques have been used in distributed file systems for data availability and stability. Typically, a method for storing data in a replication technique-based distributed file system is used, but due to the problem of space efficiency, an erasure-coding (EC) technique has been utilized mor...

Full description

Saved in:
Bibliographic Details
Published inApplied sciences Vol. 11; no. 8; p. 3298
Main Author Kim, Jeong-Joon
Format Journal Article
LanguageEnglish
Published Basel MDPI AG 01.04.2021
Subjects
Big Data
Block layout
Codes
Coding
combining
Cost recovery
Decoding
distributed storage system
Efficiency
erasure coding
exascale
Fault tolerance
Load
Load distribution
Performance degradation
recovery
Replication
Storage systems
Stress concentration
Online AccessGet full text

Cover

More Information
Summary:Various techniques have been used in distributed file systems for data availability and stability. Typically, a method for storing data in a replication technique-based distributed file system is used, but due to the problem of space efficiency, an erasure-coding (EC) technique has been utilized more recently. The EC technique improves the space efficiency problem more than the replication technique does. However, the EC technique has various performance degradation factors, such as encoding and decoding and input and output (I/O) degradation. Thus, this study proposes a buffering and combining technique in which various I/O requests that occurred during encoding in an EC-based distributed file system are combined into one and processed. In addition, it proposes four recovery measures (disk input/output load distribution, random block layout, multi-thread-based parallel recovery, and matrix recycle technique) to distribute the disk input/output loads generated during decoding.
ISSN:2076-3417
2076-3417
DOI:10.3390/app11083298