Automatic schema upgrade in a big data pipeline

Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising receiving an event that includes a plurality of fields and extracting needed fields from the plurality of fields. The method then serializes the plurality of fields...

Full description

Saved in:
Bibliographic Details
Main Authors Pippin, Michael, Willcox, David, Aleksandrovich, George, Watfa, Allie K
Format Patent
LanguageEnglish
Published 05.12.2023
Subjects
CALCULATING
COMPUTING
COUNTING
ELECTRIC DIGITAL DATA PROCESSING
PHYSICS
Online AccessGet full text

Cover

More Information
Summary:Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising receiving an event that includes a plurality of fields and extracting needed fields from the plurality of fields. The method then serializes the plurality of fields and generates a new event that includes the set of needed fields and a hidden field, the value of the hidden field comprising the serialized fields. The method then transmits the new event for processing using at least one processing stage of a stream processor. In response, the method reserializes a processed event generated by the stream processor and outputs the reserialized event to a downstream consumer.
Bibliography:Application Number: US202016881982