IceProd 2 Usage Experience

IceProd is a data processing and management framework developed by the IceCube Neutrino Observatory for processing of Monte Carlo simulations, detector data, and data driven analysis. It runs as a separate layer on top of grid and batch systems. This is accomplished by a set of daemons which process...

Full description

Saved in:
Bibliographic Details
Published inJournal of physics. Conference series Vol. 898; no. 9; pp. 92017 - 92020
Main Authors Delventhal, D, Schultz, D, Diaz Velez, J C
Format Journal Article
LanguageEnglish
Published Bristol IOP Publishing 01.10.2017
Subjects
Computational grids
Computer networks
Data analysis
Data processing
Distributed processing
Neutrinos
Physics
Workflow
Online AccessGet full text

Cover

More Information
Summary:IceProd is a data processing and management framework developed by the IceCube Neutrino Observatory for processing of Monte Carlo simulations, detector data, and data driven analysis. It runs as a separate layer on top of grid and batch systems. This is accomplished by a set of daemons which process job workflow, maintaining configuration and status information on the job before, during, and after processing. IceProd can also manage complex workflow DAGs across distributed computing grids in order to optimize usage of resources. IceProd has recently been rewritten to increase its scaling capabilities, handle user analysis workflows together with simulation production, and facilitate the integration with 3rd party scheduling tools. IceProd 2, the second generation of IceProd, has been running in production for several months now. We share our experience setting up the system and things we've learned along the way.
ISSN:1742-6588
1742-6596
DOI:10.1088/1742-6596/898/9/092017