The HSF Conditions Database Reference Implementation

• Published in: EPJ Web of conferences Vol. 295; p. 1051
• Main Authors: Mashinistov, Ruslan, Gerlach, Lino, Laycock, Paul, Formica, Andrea, Govi, Giacomo, Pinkenburg, Chris
• Format: Journal Article Conference Proceeding
• Language: English
• Published: Les Ulis EDP Sciences 2024
• Subjects: Best practice; Data processing; Distributed processing; Heterogeneous structure; Payloads; Redundancy; Relational data bases; Relocation; Storage
• ISSN: 2100-014X; 2101-6275; 2100-014X
• DOI: 10.1051/epjconf/202429501051

Conditions data is the subset of non-event data that is necessary to process event data. It poses a unique set of challenges, namely a heterogeneous structure and high access rates by distributed computing. The HSF Conditions Databases activity is a forum for cross-experiment discussions inviting as broad a participation as possible. It grew out of the HSF Community White Paper work to study conditions data access, where experts from ATLAS, Belle II, and CMS converged on a common language and proposed a schema that represents best practice. Following discussions with a broader community, including NP as well as HEP experiments, a core set of use cases, functionality and behaviour was defined with the aim to describe a core conditions database API. This paper will describe the reference implementation of both the conditions database service and the client which together encapsulate HSF best practice conditions data handling. Django was chosen for the service implementation, which uses an ORM instead of the direct use of SQL for all but one method. The simple relational database schema to organise conditions data is implemented in PostgreSQL. The task of storing conditions data payloads themselves is outsourced to any POSIX-compliant filesystem, allowing for transparent relocation and redundancy. Crucially this design provides a clear separation between retrieving the metadata describing which conditions data are needed for a data processing job, and retrieving the actual payloads from storage. The service deployment using Helm on OKD will be described together with scaling tests and operations experience from the sPHENIX experiment running more than 25k cores at BNL.