Real-Time LSM-Trees for HTAP Workloads

Real-time analytics systems employ hybrid data layouts in which data are stored in different formats throughout their lifecycle. Recent data are stored in a row-oriented format to serve OLTP workloads and support high insert rates, while older data are transformed to a column-oriented format for OLA...

Full description

Saved in:
Bibliographic Details
Published in2023 IEEE 39th International Conference on Data Engineering (ICDE) pp. 1208 - 1220
Main Authors Saxena, Hemant, Golab, Lukasz, Idreos, Stratos, Ilyas, Ihab F.
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.04.2023
Subjects
Costs
Data engineering
HTAP
Laser modes
Layout
LSM Trees
Prototypes
Real-time systems
Storage
Throughput
Online AccessGet full text

Cover

More Information
Summary:Real-time analytics systems employ hybrid data layouts in which data are stored in different formats throughout their lifecycle. Recent data are stored in a row-oriented format to serve OLTP workloads and support high insert rates, while older data are transformed to a column-oriented format for OLAP access patterns. We observe that a Log-Structured Merge (LSM) Tree is a natural fit for a lifecycle-aware storage engine due to its high write throughput and level-oriented structure, in which records propagate from one level to the next over time. To build a lifecycle-aware storage engine using an LSM-Tree, we make a crucial modification to allow different data layouts in different levels, ranging from purely row-oriented to purely column-oriented, leading to a Real-Time LSM-Tree. We give a cost model and an algorithm to design a Real-Time LSM-Tree that is suitable for a given workload, followed by an experimental evaluation of LASER - a prototype implementation of our idea built on top of the RocksDB key-value store.
ISSN:2375-026X
DOI:10.1109/ICDE55515.2023.00097