Duplicate detection and replay to ensure exactly-once delivery in a streaming pipeline

Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising processing a plurality of events using a stream processor and executing a deduplication process on the plurality of events using the stream processor. The plurality...

Full description

Saved in:
Bibliographic Details
Main Authors Pippin, Michael, Willcox, David, Aleksandrovich, George, Watfa, Allie K
Format Patent
LanguageEnglish
Published 13.08.2024
Subjects
CALCULATING
COMPUTING
COUNTING
ELECTRIC DIGITAL DATA PROCESSING
PHYSICS
Online AccessGet full text

Cover

More Information
Summary:Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising processing a plurality of events using a stream processor and executing a deduplication process on the plurality of events using the stream processor. The plurality of events is outputted to a streaming queue and a close of books (COB) of a data transport is detected. Then, an audit process is initiated in response to detecting the COB signal, the audit process comprising comparing a set of raw events to a set of events in the streaming queue to identify a set of missing events, and replaying a set of missing events through the stream processor.
Bibliography:Application Number: US202016881631