Fatman： Building Reliable Archival Storage Based on Low-Cost Volunteer Resources

• Published in: Journal of computer science and technology Vol. 30; no. 2; pp. 273 - 282
• Main Author: 覃安 胡殿明 刘俊 杨文君 谭待
• Format: Journal Article
• Language: English
• Published: Boston Springer US 01.03.2015; Springer Nature B.V
• Subjects: Analysis; Archives & records; Artificial Intelligence; Bandwidths; Budgeting; Cloud computing; Computer Science; Construction; Cost engineering; Crashes; Cutting; Data management; Data Structures and Information Theory; Design; Disks; Failure; Information Systems Applications (incl.Internet); Internet; Metadata; Network storage; Nodes; Regular Paper; Resource utilization; Servers; Software Engineering; Storage capacity; Studies; Systems design; Theory of Computation; Web服务器; 低成本; 存储容量; 存储资源; 故障恢复; 档案存储; 档案系统; 资源利用率; archival storage; failure recovery; volunteer storage; failure prediction; reliability
• ISSN: 1000-9000; 1860-4749
• DOI: 10.1007/s11390-015-1521-6

We present Fatman, an enterprise-scale archival storage based on volunteer contribution resources from underutilized web servers, usually deployed on thousands of nodes with spare storage capacity. Fatman is specifically designed for enhancing the utilization of existing storage resources and cutting down the hardware purchase cost. Two major concerned issues of the system design are maximizing the resource utilization of volunteer nodes without violating service level objectives （SLOs） and minimizing the cost without reducing the availability of archival system. Fatman has been widely deployed on tens of thousands of server nodes across several datacenters, providing more than 100 PB storage capacity and serving dozens of internal mass-data applications. The system realizes an efficient storage quota consolidation by strong isolation and budget limitation, to maximally support resource contribution without any degradation on host-level SLOs. It novelly improves data reliability by applying disk failure prediction to minish failure recovery cost, named fault-aware data management, dramatically reduces the mean time to repair （MTTR） by 76.3% and decreases file crash ratio by 35% on real-life product workload.