Data-oriented scheduling for PROOF

• Published in: Journal of physics. Conference series Vol. 331; no. 1; pp. 012009 - 7
• Main Authors: Xu, Neng, Guan, Wen, Wu, Sau Lan, Ganis, Gerardo
• Format: Journal Article
• Language: English
• Published: Bristol IOP Publishing 23.12.2011
• Subjects: Cloud computing; Computer simulation; Dynamical systems; Geographical distribution; Ingredients; Input output analysis; Physics; Policies; Priorities; Proving; Scheduling; Tasks; Virtual organization
• ISSN: 1742-6596; 1742-6588; 1742-6596
• DOI: 10.1088/1742-6596/331/1/012009

The Parallel ROOT Facility - PROOF - is a distributed analysis system optimized for I/O intensive analysis tasks of HEP data. With LHC entering the analysis phase, PROOF has become a natural ingredient for computing farms at Tier3 level. These analysis facilities will typically be used by a few tenths of users, and can also be federated into a sort of analysis cloud corresponding to the Virtual Organization of the experiment. Proper scheduling is required to guarantee fair resource usage, to enforce priority policies and to optimize the throughput. In this paper we discuss an advanced priority system that we are developing for PROOF. The system has been designed to automatically adapt to unknown length of the tasks, to take into account the data location and availability (including distribution across geographically separated sites), and the {group, user} default priorities. In this system, every element - user, group, dataset, job slot and storage - gets its priority and those priorities are dynamically linked with each other. In order to tune the interplay between the various components, we have designed and started implementing a simulation application that can model various type and size of PROOF clusters. In this application a monitoring package records all the changes of them so that we can easily understand and tune the performance. We will discuss the status of our simulation and show examples of the results we are expecting from it.