Improving Inference Performance of Machine Learning with the Divide-and-Conquer Principle

Many popular machine learning models scale poorly when deployed on CPUs. In this paper we explore the reasons why and propose a simple, yet effective approach based on the well-known Divide-and-Conquer Principle to tackle this problem of great practical importance. Given an inference job, instead of...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Author Kogan, Alex
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 01.03.2023
Subjects
Inference
Machine learning
Natural language processing
Optical character recognition
Principles
Online AccessGet full text

Cover

More Information
Summary:Many popular machine learning models scale poorly when deployed on CPUs. In this paper we explore the reasons why and propose a simple, yet effective approach based on the well-known Divide-and-Conquer Principle to tackle this problem of great practical importance. Given an inference job, instead of using all available computing resources (i.e., CPU cores) for running it, the idea is to break the job into independent parts that can be executed in parallel, each with the number of cores according to its expected computational cost. We implement this idea in the popular OnnxRuntime framework and evaluate its effectiveness with several use cases, including the well-known models for optical character recognition (PaddleOCR) and natural language processing (BERT).
ISSN:2331-8422