Research and implementation of scalable parallel computing based on Map-Reduce

• Published in: Journal of Shanghai University Vol. 15; no. 5; pp. 426 - 429
• Main Author: 阮青强 沈文枫 柴亚辉 徐炜民
• Format: Journal Article
• Language: English
• Published: Heidelberg Shanghai University Press 01.10.2011; School of Computer Engineering and Science, Shanghai University, Shanghai 200072, P.R.China%School of Computer Engineering and Science, Shanghai University, Shanghai 200072, P.R.China; School of Information Engineering, East China Jiaotong University, Nanchang 330013, P.R.China
• Subjects: Classical Mechanics; Computation; Computer programs; Encapsulation; Engineering; Environment; Fault tolerance; Information Technology; Life Sciences; Materials Science; Mathematical and Computational Engineering; Mechatronics; Parallel processing; Parallel programming; Platforms; Software; 可扩展; 地图; 容错处理; 并行编程; 并行计算方法; 海量数据; 编程模型; 负载均衡; body problem; Map-Reduce; distributed computing; N-body problem
• ISSN: 1007-6417; 1863-236X
• DOI: 10.1007/s11741-011-0763-3

As a parallel programming model, Map-Reduce is used for distributed computing of massive data. Map-Reduce model encapsulates the details of parallel implementation, fault-tolerant processing, local computing and load balancing, etc., provides a simple but powerful interface. In case of having no clear idea about distributed and parallel programming, this interface can be utilized to save development time. This paper introduces the method of using Hadoop, the open-source Map-Reduce software platform, to combine PCs to carry out scalable parallel computing. Our experiment using 12 PCs to compute N-body problem based on Map-Reduce model shows that we can get a 9.8x speedup ratio. This work indicates that the Map-Reduce can be applied in scalable parallel computing.