Leveraging Data Deduplication to Improve the Performance of Primary Storage Systems in the Cloud

• Published in: IEEE transactions on computers Vol. 65; no. 6; pp. 1775 - 1788
• Main Authors: Mao, Bo, Jiang, Hong, Wu, Suzhen, Tian, Lei
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.06.2016; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Big data; Cache storage; Clouds; Data Redundancy; Disks; Fragmentation; I/O Deduplication; I/O Performance; Indexes; Input output; Memory management; Performance evaluation; Primary Storage; Prototypes; Redundancy; Servers; Storage Capacity; Storage systems; Traffic congestion; Traffic engineering; Traffic flow; primary storage; data redundancy; I/O performance; storage capacity; I/O deduplication
• ISSN: 0018-9340; 1557-9956
• DOI: 10.1109/TC.2015.2455979

With the explosive growth in data volume, the I/O bottleneck has become an increasingly daunting challenge for big data analytics in the Cloud. Recent studies have shown that moderate to high data redundancy clearly exists in primary storage systems in the Cloud. Our experimental studies reveal that data redundancy exhibits a much higher level of intensity on the I/O path than that on disks due to relatively high temporal access locality associated with small I/O requests to redundant data. Moreover, directly applying data deduplication to primary storage systems in the Cloud will likely cause space contention in memory and data fragmentation on disks. Based on these observations, we propose a performance-oriented I/O deduplication, called POD, rather than a capacity-oriented I/O deduplication, exemplified by iDedup, to improve the I/O performance of primary storage systems in the Cloud without sacrificing capacity savings of the latter. POD takes a two-pronged approach to improving the performance of primary storage systems and minimizing performance overhead of deduplication, namely, a request-based selective deduplication technique, called Select-Dedupe, to alleviate the data fragmentation and an adaptive memory management scheme, called iCache, to ease the memory contention between the bursty read traffic and the bursty write traffic. We have implemented a prototype of POD as a module in the Linux operating system. The experiments conducted on our lightweight prototype implementation of POD show that POD significantly outperforms iDedup in the I/O performance measure by up to 87.9 percent with an average of 58.8 percent. Moreover, our evaluation results also show that POD achieves comparable or better capacity savings than iDedup.