Data stream processing check point file storage method and device based on message queue

The invention provides a data stream processing check point file storage method and device based on a message queue, and the method comprises the steps: carrying out the feature analysis according toa check point file which is generated in streaming calculation and is used for storing an intermediat...

Full description

Saved in:
Bibliographic Details
Main Authors YU GE, NIE TIEZHENG, HAO CHANGBO, SHEN DERONG, KOU YUE
Format Patent
LanguageChinese
English
Published 08.01.2021
Subjects
CALCULATING
COMPUTING
COUNTING
ELECTRIC DIGITAL DATA PROCESSING
PHYSICS
Online AccessGet full text

Cover

More Information
Summary:The invention provides a data stream processing check point file storage method and device based on a message queue, and the method comprises the steps: carrying out the feature analysis according toa check point file which is generated in streaming calculation and is used for storing an intermediate state, and carrying out the logic adaption of a storage structure of the check point file with astorage model of the message queue; designing a mapping relationship between a file operation interface and a message operation interface, and storing a mapping relationship meta-information part by using a Zookeeper; performing self-defining of a distributed file system protocol, depending a service-free form for providing a dependency library on by an upstream streaming processing application, routing check point data to a corresponding message queue storage structure according to an implemented mapping rule in a running mode transparent to a user, and when the streaming application is restarted, obtaining correspond
Bibliography:Application Number: CN202011146473