Characterizing and profiling scientific workflows

• Published in: Future generation computer systems Vol. 29; no. 3; pp. 682 - 692
• Main Authors: Juve, Gideon, Chervenak, Ann, Deelman, Ewa, Bharathi, Shishir, Mehta, Gaurang, Vahi, Karan
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.03.2013
• Subjects: Measurement; Profiling; Scientific workflows; Measurement; Scientific workflows; Profiling; Performance
• ISSN: 0167-739X; 1872-7115
• DOI: 10.1016/j.future.2012.08.015

Researchers working on the planning, scheduling, and execution of scientific workflows need access to a wide variety of scientific workflows to evaluate the performance of their implementations. This paper provides a characterization of workflows from six diverse scientific applications, including astronomy, bioinformatics, earthquake science, and gravitational-wave physics. The characterization is based on novel workflow profiling tools that provide detailed information about the various computational tasks that are present in the workflow. This information includes I/O, memory and computational characteristics. Although the workflows are diverse, there is evidence that each workflow has a job type that consumes the most amount of runtime. The study also uncovered inefficiency in a workflow component implementation, where the component was re-reading the same data multiple times. ► The workflows of six diverse scientific applications are characterized. ► The characterization includes workflow structure as well as I/O, memory and CPU usage. ► We describe new techniques that were developed to profile scientific workflows. ► The information provided can be used to create realistic synthetic workflows for use in simulation studies of workflow systems.