SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

• Published in: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS) pp. 1507 - 1512
• Main Authors: Matri, Pierre, Carns, Philip, Ross, Robert, Costan, Alexandru, Perez, Maria S., Antoniu, Gabriel
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.07.2018
• Subjects: big data; Computational modeling; Concurrent computing; convergence; Data structures; file systems; hpc; i/o; latency; Middleware; Parallel processing; replication; small files; storage; Supercomputers; Telemetry; throughput

Cloud developers traditionally rely on purpose-specific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems oft...