Efficient Data Staging Using Performance-Based Adaptation and Policy-Based Resource Allocation

Before scientific analyses run on shared infrastructure, such as the Open Science Grid or XSEDE, scientists must often transfer or stage key data sets those resources. Often these datasets consist of many files that may be transferred by multiple clients in parallel. We study two techniques that imp...

Full description

Saved in:
Bibliographic Details
Published in2014 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing pp. 244 - 247
Main Authors Chervenak, Ann L., Sim, Alex, Junmin Gu, Schuler, Robert, Hirpathak, Nandan
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.02.2014
Subjects
Concurrent computing
Conferences
Data transfer
Distributed databases
Organizations
performance-based adaptation
policy-based resource allocation
Resource management
Throughput
Online AccessGet full text

Cover

More Information
Summary:Before scientific analyses run on shared infrastructure, such as the Open Science Grid or XSEDE, scientists must often transfer or stage key data sets those resources. Often these datasets consist of many files that may be transferred by multiple clients in parallel. We study two techniques that improve the use of available resources for these large, long-running, multi-file transfers. First, we adapt transfer parameters for multi-file transfers based on recent transfer performance. Second, we use VO and site policies to influence the allocation of system resources for transfers, such as available transfer streams. We describe our system design and summarize its implementation and performance.
ISSN:1066-6192
2377-5750
DOI:10.1109/PDP.2014.49