Data stream processing check point file storage method and device based on message queue

• Main Authors: YU GE, NIE TIEZHENG, HAO CHANGBO, SHEN DERONG, KOU YUE
• Format: Patent
• Language: Chinese; English
• Published: 08.01.2021
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

The invention provides a data stream processing check point file storage method and device based on a message queue, and the method comprises the steps: carrying out the feature analysis according toa check point file which is generated in streaming calculation and is used for storing an intermediate state, and carrying out the logic adaption of a storage structure of the check point file with astorage model of the message queue; designing a mapping relationship between a file operation interface and a message operation interface, and storing a mapping relationship meta-information part by using a Zookeeper; performing self-defining of a distributed file system protocol, depending a service-free form for providing a dependency library on by an upstream streaming processing application, routing check point data to a corresponding message queue storage structure according to an implemented mapping rule in a running mode transparent to a user, and when the streaming application is restarted, obtaining correspond