Data processing method and apparatus for data skew

• Main Authors: CHEN JIANZHANG, JI YINGLONG
• Format: Patent
• Language: Chinese; English
• Published: 03.01.2023
• Subjects: CALCULATING; COMPUTING; COUNTING; DATA PROCESSING SYSTEMS OR METHODS, SPECIALLY ADAPTED FORADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORYOR FORECASTING PURPOSES; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE,COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTINGPURPOSES, NOT OTHERWISE PROVIDED FOR

The invention relates to a data processing method and device for data skew, and the method comprises the steps: obtaining to-be-processed associated data in a key-value pair form, the key and value of the associated data being respectively corresponding to an identifier of a processing object and detail data of the processing object; pre-judging whether a key corresponding to the associated data is a hotspot key or not, wherein the hotspot key is used for representing the state that computing resources needing to be consumed by one-time computing of detail data of the associated data exceed a set threshold value; under the condition that the key corresponding to the associated data is pre-judged to be a hotspot key, performing initial grouping on the associated data to enable the data processing amount of each group to accord with a preset range, and distributing each group of divided associated data to a resource slot of a first processing cluster for calculation to obtain an intermediate processing result;