Efficient execution of the WRF model and other HPC applications in the cloud

• Published in: Earth science informatics Vol. 9; no. 3; pp. 365 - 382
• Main Authors: Duran-Limon, Hector A., Flores-Contreras, Jesus, Parlavantzas, Nikos, Zhao, Ming, Meulenert-Peña, Angel
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.09.2016; Springer Nature B.V; Springer Link
• Subjects: Cloud computing; Computation; Computer Science; Demand; Distributed, Parallel, and Cluster Computing; Earth and Environmental Science; Earth Sciences; Earth System Sciences; High performance computing; Information Systems Applications (incl.Internet); Lightweight; Mathematical models; Methodology Article; Ontology; Platforms; Running; Science; Simulation and Modeling; Space Exploration and Astronautics; Space Sciences (including Extraterrestrial Physics; Weight reduction; WRF model; Cloud computing; Lightweight virtual machines; Virtualisation; High performance computing
• ISSN: 1865-0473; 1865-0481
• DOI: 10.1007/s12145-016-0253-7

There are many scientific applications that have high performance computing (HPC) demands. Such demands are traditionally supported by cluster- or Grid-based systems. Cloud computing, which has experienced a tremendous growth, emerged as an approach to provide on-demand access to computing resources. The cloud computing paradigm offers a number of advantages over other distributed platforms. For example, the access to resources is flexible and cost-effective since it is not necessary to invest a large amount of money on a computing infrastructure nor pay salaries for maintenance functions. Therefore, the possibility of using cloud computing for running high performance computing applications is attractive. However, it has been shown elsewhere that current cloud computing platforms are not suitable for running some of these kinds of applications since the performance offered is very poor. The reason is mainly the overhead from virtualisation which is extensively used by most cloud computing platforms as a means to optimise resource usage. Furthermore, running HPC applications in current cloud platforms is a complex task that in many cases requires configuring a cluster of virtual machines (VMs). In this paper, we present a lightweight virtualisation approach for efficiently running the Weather Research and Forecasting (WRF) model (a computing- and communication-intensive application) in a cloud computing environment. Our approach also provides a higher-level programming model that automates the process of configuring a cluster of VMs. We assume such a cloud environment can be shared with other types of HPC applications such as mpiBLAST (an embarrassingly parallel application), and MiniFE (a memory-intensive application). Our experimental results show that lightweight virtualisation imposes about 5 % overhead and it substantially outperforms traditional heavyweight virtualisation such as KVM.