Data synchronization method, system and device and storage medium

• Main Authors: ZHANG YIBIN, LIANG YUBO, CHEN XIONGFENG
• Format: Patent
• Language: Chinese; English
• Published: 05.04.2024
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

The invention provides a data synchronization method, system and device and a storage medium, and the method comprises the steps: analyzing task parameters of a to-be-synchronized task, and obtaining information of a source end database, information of a source end table, information of a data warehouse end table, a filtering condition and data table mapping information, the Kafka data reading component obtains to-be-synchronized data from a source end table according to information of a source end database and information of the source end table, the to-be-synchronized data is stored in a theme message queue, the data filtering component filters the to-be-synchronized data in the theme message queue on the basis of filtering conditions, and the to-be-synchronized data is stored in the source end table. And the data synchronization component writes the filtered to-be-synchronized data into the data warehouse end table, and synchronizes the to-be-synchronized data in the data warehouse end table to the data wa