A Cross-Layer Solution in Scientific Workflow System for Tackling Data Movement Challenge
Scientific applications in HPC environment are more com-plex and more data-intensive nowadays. Scientists usually rely on workflow system to manage the complexity: simply define multiple processing steps into a single script and let the work-flow systems compile it and schedule all tasks accordingly...
Saved in:
Published in | arXiv.org |
---|---|
Main Authors | , , , , , , |
Format | Paper |
Language | English |
Published |
Ithaca
Cornell University Library, arXiv.org
16.05.2018
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | Scientific applications in HPC environment are more com-plex and more data-intensive nowadays. Scientists usually rely on workflow system to manage the complexity: simply define multiple processing steps into a single script and let the work-flow systems compile it and schedule all tasks accordingly. Numerous workflow systems have been proposed and widely used, like Galaxy, Pegasus, Taverna, Kepler, Swift, AWE, etc., to name a few examples. Traditionally, scientific workflow systems work with parallel file systems, like Lustre, PVFS, Ceph, or other forms of remote shared storage systems. As such, the data (including the intermediate data generated during workflow execution) need to be transferred back and forth between compute nodes and storage systems, which introduces a significant performance bottleneck on I/O operations. Along with the enlarging perfor-mance gap between CPU and storage devices, this bottleneck is expected to be worse. Recently, we have introduced a new concept of Compute-on-Data-Path to allow tasks and data binding to be more efficient to reduce the data movement cost. To workflow systems, the key is to exploit the data locality in HPC storage hierarchy: if the datasets are stored in compute nodes, near the workflow tasks, then the task can directly access them with better performance with less network usage. Several recent studies have been done regarding building such a shared storage system, utilizing compute node resources, to serve HPC workflows with locality, such as Hercules [1] and WOSS [2] etc. In this research, we further argue that providing a compute-node side storage system is not sufficient to fully exploit data locality. A cross-layer solution combining storage system, compiler, and runtime is necessary. We take Swift/T [3], a workflow system for data-intensive applications, as a prototype platform to demonstrate such a cross-layer solution |
---|---|
AbstractList | Scientific applications in HPC environment are more com-plex and more data-intensive nowadays. Scientists usually rely on workflow system to manage the complexity: simply define multiple processing steps into a single script and let the work-flow systems compile it and schedule all tasks accordingly. Numerous workflow systems have been proposed and widely used, like Galaxy, Pegasus, Taverna, Kepler, Swift, AWE, etc., to name a few examples. Traditionally, scientific workflow systems work with parallel file systems, like Lustre, PVFS, Ceph, or other forms of remote shared storage systems. As such, the data (including the intermediate data generated during workflow execution) need to be transferred back and forth between compute nodes and storage systems, which introduces a significant performance bottleneck on I/O operations. Along with the enlarging perfor-mance gap between CPU and storage devices, this bottleneck is expected to be worse. Recently, we have introduced a new concept of Compute-on-Data-Path to allow tasks and data binding to be more efficient to reduce the data movement cost. To workflow systems, the key is to exploit the data locality in HPC storage hierarchy: if the datasets are stored in compute nodes, near the workflow tasks, then the task can directly access them with better performance with less network usage. Several recent studies have been done regarding building such a shared storage system, utilizing compute node resources, to serve HPC workflows with locality, such as Hercules [1] and WOSS [2] etc. In this research, we further argue that providing a compute-node side storage system is not sufficient to fully exploit data locality. A cross-layer solution combining storage system, compiler, and runtime is necessary. We take Swift/T [3], a workflow system for data-intensive applications, as a prototype platform to demonstrate such a cross-layer solution |
Author | Dai, Dong Dorier, Matthieu Khaldi, Dounia Yan, Yonghong Ross, Robert Chen, Yong Tavakoli, Neda |
Author_xml | – sequence: 1 givenname: Dong surname: Dai fullname: Dai, Dong – sequence: 2 givenname: Robert surname: Ross fullname: Ross, Robert – sequence: 3 givenname: Dounia surname: Khaldi fullname: Khaldi, Dounia – sequence: 4 givenname: Yonghong surname: Yan fullname: Yan, Yonghong – sequence: 5 givenname: Matthieu surname: Dorier fullname: Dorier, Matthieu – sequence: 6 givenname: Neda surname: Tavakoli fullname: Tavakoli, Neda – sequence: 7 givenname: Yong surname: Chen fullname: Chen, Yong |
BookMark | eNqNyr0KwjAUQOEgCtafd7jgXGiT1uooVXHQqYI4SSi3GpvmapIqfXsdfACnM5xvxPqGDPZYwIWIw0XC-ZBNnbtHUcTnGU9TEbDzCnJLzoV72aGFgnTrFRlQBopSofGqUiWcyNaVpjcUnfPYQEUWjrKstTJXWEsv4UAvbL4c8pvUGs0VJ2xQSe1w-uuYzbabY74LH5aeLTp_uVNrzXddeJSJbBkn81j8pz4Ps0OS |
ContentType | Paper |
Copyright | 2018. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
Copyright_xml | – notice: 2018. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
DBID | 8FE 8FG ABJCF ABUWG AFKRA AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ L6V M7S PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
DatabaseName | ProQuest SciTech Collection ProQuest Technology Collection Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest Central ProQuest Central Essentials AUTh Library subscriptions: ProQuest Central Technology Collection ProQuest One Community College ProQuest Central SciTech Premium Collection (Proquest) (PQ_SDU_P3) ProQuest Engineering Collection Engineering Database Publicly Available Content Database ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection |
DatabaseTitle | Publicly Available Content Database Engineering Database Technology Collection ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central China ProQuest Central ProQuest Engineering Collection ProQuest One Academic UKI Edition ProQuest Central Korea Materials Science & Engineering Collection ProQuest One Academic Engineering Collection |
DatabaseTitleList | Publicly Available Content Database |
Database_xml | – sequence: 1 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Physics |
EISSN | 2331-8422 |
Genre | Working Paper/Pre-Print |
GroupedDBID | 8FE 8FG ABJCF ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BENPR BGLVJ CCPQU DWQXO FRJ HCIFZ L6V M7S M~E PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
ID | FETCH-proquest_journals_20737914613 |
IEDL.DBID | BENPR |
IngestDate | Tue Sep 24 21:21:40 EDT 2024 |
IsOpenAccess | true |
IsPeerReviewed | false |
IsScholarly | false |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-proquest_journals_20737914613 |
OpenAccessLink | https://www.proquest.com/docview/2073791461/abstract/?pq-origsite=%requestingapplication% |
PQID | 2073791461 |
PQPubID | 2050157 |
ParticipantIDs | proquest_journals_2073791461 |
PublicationCentury | 2000 |
PublicationDate | 20180516 |
PublicationDateYYYYMMDD | 2018-05-16 |
PublicationDate_xml | – month: 05 year: 2018 text: 20180516 day: 16 |
PublicationDecade | 2010 |
PublicationPlace | Ithaca |
PublicationPlace_xml | – name: Ithaca |
PublicationTitle | arXiv.org |
PublicationYear | 2018 |
Publisher | Cornell University Library, arXiv.org |
Publisher_xml | – name: Cornell University Library, arXiv.org |
SSID | ssj0002672553 |
Score | 3.1360528 |
SecondaryResourceType | preprint |
Snippet | Scientific applications in HPC environment are more com-plex and more data-intensive nowadays. Scientists usually rely on workflow system to manage the... |
SourceID | proquest |
SourceType | Aggregation Database |
SubjectTerms | Galaxies Luster Nodes Schedules Servers Storage systems Task scheduling Workflow Workflow management systems |
Title | A Cross-Layer Solution in Scientific Workflow System for Tackling Data Movement Challenge |
URI | https://www.proquest.com/docview/2073791461/abstract/ |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1bS8MwFD5sK4JvXvEyR0Bfw3pJ0_ZJdLYWcWPIhPk0kjQVYbRzq_jmbzentvog7DEEQhKSc7585-R8AFdBJLmTu5xGKlOUSc6pZMKh2pwVxQJlIDc-FMcTnj6zh7k_70Da_oXBtMrWJtaGOisVcuTIhHhBhCrUQyGRBVDV8Hr1TlE_CuOsjZhGFyzXYRiwtW7jyfTpl29xeWDQs_fP5NZ-JNkDaypWer0PHV0cwE6dfqk2h_ByQ0boruijMBCYtFwVeStIffnqhB6CxHa-LD_JT5lxYvAmmQmFmuuv5E5UgozLuvx3RUatRsoRXCbxbJTSdj6L5vRsFn9r9Y6hV5SFPgESZb6nWeib94BmuZZCczsKbFtntg59aZ9Cf9tIZ9u7z2HXQIEQ4-IO70OvWn_oC-NuKzmAbpjcD5r9NK3xV_wNrEqLQQ |
link.rule.ids | 786,790,12792,21416,33408,33779,43635,43840 |
linkProvider | ProQuest |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1NS8QwEB10F9Gbn_ix6oBeA-02TduTSHWt2i4eKqynkqSpCLJdtxX_vkls9SDsORCSkMy8eTOZB3AZRIK51ZiRSJaSUMEYEZS7ROm7ImkgNeQ2gWI2ZckzfZj5s45wa7qyyt4mWkNd1tJw5IYJ8YLIqFBfLT6IUY0y2dVOQmMdhtTTrtP8FJ_c_XIsYxZoxOz9M7PWd0y2YfjEF2q5A2tqvgsbtuRSNnvwco2xcVEk5Rr2Ys9P4dsc7YOzRTxoyOzqvf7Cn9biqDEm5lwanfVXvOEtx6y2Lb9bjHtdlH24mNzmcUL69RTdjWmKv_15BzDQob86BIxK31M09HUMoGilBFfMiQLHUaWjQl84RzBaNdPx6uFz2EzyLC3S--njCWxpKBCavLjLRjBol5_qVLvbVpzZM_0G3BmG9w |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Cross-Layer+Solution+in+Scientific+Workflow+System+for+Tackling+Data+Movement+Challenge&rft.jtitle=arXiv.org&rft.au=Dai%2C+Dong&rft.au=Ross%2C+Robert&rft.au=Khaldi%2C+Dounia&rft.au=Yan%2C+Yonghong&rft.date=2018-05-16&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422 |