Spark SQL Shuffle task number optimization system based on historical information

• Main Authors: ZHAO ZHIFENG, CAO JUNLIANG, LONG YILIN, WANG YONGQIANG, WANG XIAODONG, CHANG YI, XIA JUNSHENG
• Format: Patent
• Language: Chinese; English
• Published: 05.04.2024
• Subjects: CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

The invention discloses a Spark SQL Shuffle task number optimization system based on historical information, and relates to the field of big data, databases and machine learning, the Spark SQL Shuffle task number optimization system comprises an SQL historical operation information extraction module...