Streamlining CASTOR to manage the LHC data torrent

• Published in: Journal of physics. Conference series Vol. 513; no. 4; pp. 42031 - 6
• Main Authors: Presti, G Lo, Curull, X Espinal, Cano, E, Fiorini, B, Ieri, A, Murray, S, Ponce, S, Sindrilaru, E
• Format: Journal Article
• Language: English
• Published: Bristol IOP Publishing 01.01.2014
• Subjects: CERN; Cold storage; Data management; Data mining; Data transmission; Evolution; Infrastructure; Monitoring; Physics; Policies; Recall; Streamlining; Torrents
• ISSN: 1742-6596; 1742-6588; 1742-6596
• DOI: 10.1088/1742-6596/513/4/042031

This contribution describes the evolution of the main CERN storage system, CASTOR, as it manages the bulk data stream of the LHC and other CERN experiments, achieving over 90 PB of stored data by the end of LHC Run 1. This evolution was marked by the introduction of policies to optimize the tape sub-system throughput, going towards a cold storage system where data placement is managed by the experiments' production managers. More efficient tape migrations and recalls have been implemented and deployed where bulk meta-data operations greatly reduce the overhead due to small files. A repack facility is now integrated in the system and it has been enhanced in order to automate the repacking of several tens of petabytes, required in 2014 in order to prepare for the next LHC run. Finally the scheduling system has been evolved to integrate the internal monitoring. To efficiently manage the service a solid monitoring infrastructure is required, able to analyze the logs produced by the different components (about 1 kHz of log messages). A new system has been developed and deployed, which uses a transport messaging layer provided by the CERN-IT Agile Infrastructure and exploits technologies including Hadoop and HBase. This enables efficient data mining by making use of MapReduce techniques, and real-time data aggregation and visualization. The outlook for the future is also presented. Directions and possible evolution will be discussed in view of the restart of data taking activities.