Dataplug: Unlocking extreme data analytics with on-the-fly dynamic partitioning of unstructured data

The elasticity of the Cloud is very appealing for processing large scientific data. However, enormous volumes of unstructured research data, totaling petabytes, remain untapped in data repositories due to the lack of efficient parallel data access. Even-sized partitioning of these data to enable its...

Full description

Saved in:
Bibliographic Details
Published in2024 IEEE 24th International Symposium on Cluster, Cloud and Internet Computing (CCGrid) pp. 567 - 576
Main Authors Arjona, Aitor, Garcia-Lopez, Pedro, Barcelona-Pons, Daniel
Format Conference Proceeding
LanguageEnglish
Published IEEE 06.05.2024
Subjects
Bandwidth
Bioinformatics
Cloud
Cloud computing
Costs
data analytics
data partitioning
Distributed databases
Dynamic scheduling
Genomics
Geospatial analysis
object storage
Scientific computing
scientific data management
Semantics
unstructured data
Online AccessGet full text

Cover

More Information
Summary:The elasticity of the Cloud is very appealing for processing large scientific data. However, enormous volumes of unstructured research data, totaling petabytes, remain untapped in data repositories due to the lack of efficient parallel data access. Even-sized partitioning of these data to enable its parallel processing requires a complete re-write to storage, becoming prohibitively expensive for high volumes. In this article we present Dataplug, an extensible framework that enables fine-grained parallel data access to unstructured scientific data in object storage. Dataplug employs read-only, format-aware indexing, allowing to define dynamically-sized partitions using various partitioning strategies. This approach avoids writing the partitioned dataset back to storage, enabling distributed workers to fetch data partitions on-the-fly directly from large data blobs, efficiently leveraging the high bandwidth capability of object storage. Validations on genomic (FASTQGZip) and geospatial (LiDAR) data formats demonstrate that Dataplug considerably lowers pre-processing compute costs (between 65.5% - 71.31% less) without imposing significant overheads.
ISSN:2993-2114
DOI:10.1109/CCGrid59990.2024.00069