ADAPTIVE MAP-REDUCE PIPELINE WITH DYNAMIC THREAD ALLOCATIONS

In an approach to adaptively pipeline a MapReduce job, a processor receives one or more data records from a storage and inserts the one or more data records into a map queue, wherein a size of the map queue is adaptive to one or more utilizations of one or more resources in the processor. One or mor...

Full description

Saved in:
Bibliographic Details
Main Authors Ma Haohai, Bao Xiao Ming, Bai Wen Yan, Hu Zhenhua, Liu Jian
Format Patent
LanguageEnglish
Published 06.10.2016
Subjects
CALCULATING
COMPUTING
COUNTING
ELECTRIC DIGITAL DATA PROCESSING
PHYSICS
Online AccessGet full text

Cover

More Information
Summary:In an approach to adaptively pipeline a MapReduce job, a processor receives one or more data records from a storage and inserts the one or more data records into a map queue, wherein a size of the map queue is adaptive to one or more utilizations of one or more resources in the processor. One or more processors apply a map function to the one or more data records in the first buffer and sort the records that are output from the map function and store the sorted records. One or more processors receive and insert the sorted records into a reduce queue, wherein a size of the reduce queue is adaptive to one or more utilizations of resources in the one or more processors. One or more processors apply a reduce function to the sorted records in the reduce queue and store a result in a storage.
Bibliography:Application Number: US201514672781