Determining optimal compute resources for distributed batch based optimization applications

• Main Authors: Radhakrishnan, Jayaram Kallapalayam, Saxena, Vaibhav, Sabharwal, Yogish, Basu, Saurav, Verma, Ashish
• Format: Patent
• Language: English
• Published: 01.03.2022
• Subjects: CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; DATA PROCESSING SYSTEMS OR METHODS, SPECIALLY ADAPTED FORADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORYOR FORECASTING PURPOSES; ELECTRIC DIGITAL DATA PROCESSING; HANDLING RECORD CARRIERS; PHYSICS; PRESENTATION OF DATA; RECOGNITION OF DATA; RECORD CARRIERS; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE,COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTINGPURPOSES, NOT OTHERWISE PROVIDED FOR

Methods, systems, and computer program products for determining optimal compute resources for distributed batch based optimization applications are provided herein. A method includes obtaining a size of an input dataset, a size of a model, and a set of batch sizes corresponding to a job to be processed using a distributed computing system; computing, based at least in part on the set of batch sizes, one or more node counts corresponding to a number of nodes that can be used for processing said job; estimating, for each given one of the node counts, an execution time to process the job based on an average computation time for a batch of said input dataset and an average communication time for said batch of said input dataset; and selecting, based at least in part on said estimating, at least one of said node counts for processing the job.