MOSAIC: Heterogeneity-, Communication-, and Constraint-Aware Model Slicing and Execution for Accurate and Efficient Inference

• Published in: Proceedings / International Conference on Parallel Architectures and Compilation Techniques pp. 165 - 177
• Main Authors: Han, Myeonggyun, Hyun, Jihoon, Park, Seongbeom, Park, Jinsu, Baek, Woongki
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.09.2019
• Subjects: Computational modeling; Embedded systems; Energy consumption; Graphics processing units; Heterogeneous Embedded Systems; Inference; Memory management; Model Slicing and Execution; Performance evaluation
• ISSN: 2641-7936
• DOI: 10.1109/PACT.2019.00021

Heterogeneous embedded systems have surfaced as a promising solution for accurate and efficient deep-learning inference on mobile devices. Despite extensive prior works, it still remains unexplored to investigate the system-software support that efficiently executes inference workloads by judiciously considering their performance and energy heterogeneity, communication overheads, and constraints. To bridge this gap, we propose MOSAIC, heterogeneity-, communication-, and constraint-aware model slicing and execution for accurate and efficient inference on heterogeneous embedded systems. MOSAIC generates the efficient model slicing and execution plan for the target inference workload through dynamic programming. MOSAIC significantly reduces inference latency and energy, exhibits high estimation accuracy, and incurs small overheads.