In-Staging Data Placement for Asynchronous Coupling of Task-Based Scientific Workflows

• Published in: 2016 Second International Workshop on Extreme Scale Programming Models and Middlewar (ESPM2) pp. 2 - 9
• Main Authors: Qian Sun, Romanus, Melissa, Tong Jin, Hongfeng Yu, Bremer, Peer-Timo, Petruzza, Steve, Klasky, Scott, Parashar, Manish
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.11.2016
• Subjects: Adaptation models; Couplings; Data models; Data processing; Data storage systems; Distributed databases; Monitoring; Runtime
• DOI: 10.1109/ESPM2.2016.006

Coupled application workflows composed of applications implemented using task-based models present new coupling and data exchange challenges, due to the asynchronous interaction and coupling behaviors between tasks of the component applications. In this paper, we present an adaptive data placement approach that addresses these challenges by dynamically adjusting to the asynchronous coupling patterns. Specifically, it places data across a set of staging cores/nodes with an awareness of the application-specific data locality requirements and the runtime task executions at these staging cores/nodes, with the goal of reducing end-to-end execution time and data movement overhead of the workflow. We experimentally demonstrate the effectiveness of our approach on the Titan Cray XK7 system using representative data coupling patterns derived from current scientific workflows. The evaluation demonstrates that our approach efficiently improves performance by reducing the time-to-solution and increasing the quality of insights for scientific discovery.