Kubernetes Container Orchestration as a Framework for Flexible and Effective Scientific Data Analysis

In this paper we present the design and deployment details for Single Particle Imaging (SPI) experiments data analysis pipeline in Kubernetes infrastructure. We have analyzed various software usage patterns for different payloads. Components of the pipeline software include traditional HPC (MPI-base...

Full description

Saved in:
Bibliographic Details
Published in2019 Ivannikov Ispras Open Conference (ISPRAS) pp. 67 - 71
Main Authors Tesliuk, Anton, Bobkov, Sergey, Ilyin, Viacheslav, Novikov, Alexander, Poyda, Alexey, Velikhov, Vasily
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.12.2019
Subjects
big data
Containers
data analysis
Graphics processing units
kubernetes
Parallel processing
Payloads
Pipelines
single particle imaging
software pipeline
Task analysis
Online AccessGet full text

Cover

More Information
Summary:In this paper we present the design and deployment details for Single Particle Imaging (SPI) experiments data analysis pipeline in Kubernetes infrastructure. We have analyzed various software usage patterns for different payloads. Components of the pipeline software include traditional HPC (MPI-based) applications, applications which require GPU computations, GUI-based software and the software which can be parallelized naturally by dividing the data into several independent parts (Data parallelism). For every payload type individual deployment approach was proposed.
DOI:10.1109/ISPRAS47671.2019.00016