ADAPTIVE MAP-REDUCE PIPELINE WITH DYNAMIC THREAD ALLOCATIONS

• Main Authors: Ma Haohai, Bao Xiao Ming, Bai Wen Yan, Hu Zhenhua, Liu Jian
• Format: Patent
• Language: English
• Published: 06.10.2016
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

In an approach to adaptively pipeline a MapReduce job, a processor receives one or more data records from a storage and inserts the one or more data records into a map queue, wherein a size of the map queue is adaptive to one or more utilizations of one or more resources in the processor. One or more processors apply a map function to the one or more data records in the first buffer and sort the records that are output from the map function and store the sorted records. One or more processors receive and insert the sorted records into a reduce queue, wherein a size of the reduce queue is adaptive to one or more utilizations of resources in the one or more processors. One or more processors apply a reduce function to the sorted records in the reduce queue and store a result in a storage.