Processing Particle Data Flows with SmartNICs

Many distributed applications implement complex data flows and need a flexible mechanism for routing data between producers and consumers. Recent advances in programmable network interface cards, or SmartNICs, represent an opportunity to offload data-flow tasks into the network fabric, thereby freei...

Full description

Saved in:
Bibliographic Details
Published in2022 IEEE High Performance Extreme Computing Conference (HPEC) pp. 1 - 8
Main Authors Liu, Jianshen, Maltzahn, Carlos, Curry, Matthew L., Ulmer, Craig
Format Conference Proceeding
LanguageEnglish
Published IEEE 19.09.2022
Subjects
Atmospheric measurements
compression
Distributed databases
Fabrics
Hardware
in-transit computations
Particle measurements
Partitioning algorithms
Routing
SmartNICs
Online AccessGet full text

Cover

More Information
Summary:Many distributed applications implement complex data flows and need a flexible mechanism for routing data between producers and consumers. Recent advances in programmable network interface cards, or SmartNICs, represent an opportunity to offload data-flow tasks into the network fabric, thereby freeing the hosts to perform other work. System architects in this space face multiple questions about the best way to leverage SmartNICs as processing elements in data flows. In this paper, we advocate the use of Apache Arrow as a foundation for implementing data-flow tasks on SmartNICs. We report on our experiences adapting a partitioning algorithm for particle data to Apache Arrow and measure the on-card processing performance for the BlueField-2 SmartNIC. Our experiments confirm that the BlueField-2's (de)compression hardware can have a significant impact on in-transit workflows where data must be unpacked, processed, and repacked.
ISSN:2643-1971
DOI:10.1109/HPEC55821.2022.9926325