Operator concurrency adjusting and optimizing method and device for stream processing system

• Main Authors: PU ZENGLIN, XIANG QINGFENG, SHAO YINGXIA, LIAN JINQING
• Format: Patent
• Language: Chinese; English
• Published: 06.06.2023
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

The invention provides an operator concurrency adjusting and optimizing method and device for a stream processing system, and the method comprises the steps: carrying out the excessive capacity expansion adjusting and optimizing of a stream processing task in a backpressure state, so as to enable each stream processing task to be in a to-be-reduced state at present, and to serve as a target stream processing task; training an abstract operator in the stream processing system according to historical tuning information to generate a GP model, and calculating to obtain a workload adaptability score and a self-design WOP acquisition function according to a historical estimated load in the GP model and a current workload of the target stream processing task; and determining the optimal concurrency of the abstract operator according to the GP model and the self-design WOP acquisition function. Compared with the prior art, the method does not depend on an adjustment and optimization built-in rule, makes full use of