Re-sizing data partitions for ensemble models in a mapreduce framework

• Main Authors: Dagli C. Ates, McCarroll Niall Fraser, Vasilenko Dmitry
• Format: Patent
• Language: English
• Published: 24.10.2017
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

Techniques are described for revising data partition size for use in generating predictive models. In one example, a method includes determining an initial number of base model partitions of data from a plurality of data sources; determining an initial base model partition size based at least in part on the initial number of base model partitions; and evaluating the initial base model partition size at least in part with reference to at least one base model partition size reference. The method further includes determining a finalized number of base model partitions based at least in part on the initial base model partition size; determining a revised base model partition size; and generating revised base models based at least in part on the revised base model partition size, including using a predictive modeling framework to randomly assign input data records from the plurality of data sources into the base model partitions.