SELF-CORRECTING PIPELINE FLOWS FOR SCHEMA DRIFT

• Main Authors: Varma, Vikas, Leo, Sathish Paul, Allan, David, Klein, Frank Joseph, Thatte, Sachin Sadashiv, Patnaik, Arun
• Format: Patent
• Language: English
• Published: 23.04.2020
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

Techniques describes herein updating pipeline flows in within data systems to maintain data integrity and consistency without manual curation. In certain embodiments, data integration system may detect and/or receive indications of a schema change within a source system of the data integration system. One or more data objects affected by the schema change may be identified, and a set of pipeline rules may be invoked for each of the affected schema changes. The pipeline rules may define a single transformation or a multi-step transformation process by which the data in the source system is provided to one or more target systems. After applying the pipeline rules to the updated source schema, the data received from the source system may be processed using the updated pipeline rules, transformed, and transmitted to the target system(s) to maintain the data integrity of the system.