RCFile: A fast and space-efficient data placement structure in MapReduce-based warehouse systems

• Published in: 2011 IEEE 27th International Conference on Data Engineering pp. 1199 - 1208
• Main Authors: Yongqiang He, Rubao Lee, Yin Huai, Zheng Shao, Jain, N, Xiaodong Zhang, Zhiwei Xu
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.04.2011
• Subjects: Data compression; Data handling; Data storage systems; Data warehouses; Facebook; Information management; Query processing
• ISBN: 9781424489596; 1424489598
• ISSN: 1063-6382; 2375-026X
• DOI: 10.1109/ICDE.2011.5767933

MapReduce-based data warehouse systems are playing important roles of supporting big data analytics to understand quickly the dynamics of user behavior trends and their needs in typical Web service providers and social network sites (e.g., Facebook). In such a system, the data placement structure is a critical factor that can affect the warehouse performance in a fundamental way. Based on our observations and analysis of Facebook production systems, we have characterized four requirements for the data placement structure: (1) fast data loading, (2) fast query processing, (3) highly efficient storage space utilization, and (4) strong adaptivity to highly dynamic workload patterns. We have examined three commonly accepted data placement structures in conventional databases, namely row-stores, column-stores, and hybrid-stores in the context of large data analysis using MapReduce. We show that they are not very suitable for big data processing in distributed systems. In this paper, we present a big data placement structure called RCFile (Record Columnar File) and its implementation in the Hadoop system. With intensive experiments, we show the effectiveness of RCFile in satisfying the four requirements. RCFile has been chosen in Facebook data warehouse system as the default option. It has also been adopted by Hive and Pig, the two most widely used data analysis systems developed in Facebook and Yahoo!