US 2024/0402990 A1Published application

Maintaining streaming parity in large-scale pipelines

AssigneeIntuit Inc.

InventorsSaikiran Sri THUNUGUNTLA, Kaushik S, Sudarma Denson POKTA, Amit Kumar SONI, Aman AGRAWAL

Life of this patent

Estimating the term…

Loading patent…

Abstract

In a pipeline, data events generated by a producer application are temporally grouped by using a group identification tag. For each data event, data points are generated and uploaded to a storage and cache at each point of production and consumption. The storage allows a matching of data events between the production point and the consumption point, thereby ensuring that streaming parity is maintained. In cases of mismatch, the cache allows for detecting missing data events, i.e., identifying data events that were generated by an upstream producer application, but not consumed by a downstream consumer. While being agnostic to the transformations applied by the various applications in the pipeline, the embodiments disclosed herein keep track of the output data events and input data events and precisely identify the missing data events.

Claims20

Dependent claims

A computer-implemented method for maintaining streaming parity of data events in a pipeline, the method comprising:

temporally grouping data events generated by a first application in the pipeline by adding a group identification tag to data events generated within a predetermined time window;
uploading a first set of data points identifying temporally grouped data events to a storage;
for the first set of data points, uploading to a cache, a first set of composite keys, each comprising the group identification tag and a first identification tag of the first application;
retrieving temporally grouped data events received by a second application from the first application;
uploading to the storage a second set of data points identifying temporally grouped data events received by the second application from the first application;
for the second set of data points, uploading to the cache, a second set of composite keys, each comprising the group identification tag and a second identification tag of the second application;
aggregating the first set of data points based on the first identification tag and the group identification tag by:
removing duplicate event identification tags associated with the first set of composite keys;
organizing remaining event identification tags according to the first set of composite keys;
aggregating the second set of data points based on the second identification tag and the group identification tag by:
removing duplicate event identification tags associated with the second set of composite keys;
organizing remaining event identification tags according to the second set of composite keys;
determining that the streaming parity of the temporally grouped data events is not maintained by determining that the aggregated first set of data points does not match the aggregated second set of data points;
identifying a data event generated by the first application but not received by the second application based on comparing the first set of composite keys and the second set of composite keys;
and
performing a remedial action to maintain the streaming parity.

13 more elements

A system comprising:

a non-transitory storage medium storing computer program instructions;
and
a processor configured to store the computer program instructions to cause operations comprising:
temporally grouping data events generated by a first application in a pipeline by adding a group identification tag to data events generated within a predetermined time window;
uploading to a storage a first set of data points identifying temporally grouped data events to a storage;
for the first set of data points, uploading to a cache, a first set of composite keys, each comprising the group identification tag and a first identification tag of the first application;
retrieving temporally grouped data events received by a second application from the first application;
uploading to the storage a second set of data points identifying temporally grouped data events received by the second application from the first application;
for the second set of data points, uploading to the cache, a second set of composite keys, each comprising the group identification tag and a second identification tag of the second application;
aggregating the first set of data points based on the first identification tag and the group identification tag by:
removing duplicate event identification tags associated with the first set of composite keys;
organizing remaining event identification tags according to the first set of composite keys;
aggregating the second set of data points based on the second identification tag and the group identification tag by:
removing duplicate event identification tags associated with the second set of composite keys;
organizing remaining event identification tags according to the second set of composite keys;
determining that a streaming parity of the temporally grouped data events is not maintained by determining that the aggregated first set of data points does not match the aggregated second set of data points;
identifying a data event generated by the first application but not received by the second application based on comparing the first set of composite keys and the second set of composite keys;
and
performing a remedial action to maintain the streaming parity.

16 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/326,893Filed 31 May 2023
Published: 5 Dec 2024
Term ends: Not estimatedThe source records don’t support an estimate.

Classification9 codes

G06F 7/02
Methods or arrangements for processing data by operating upon the order or content of the data handled: comparing digital values
G06F 9/4881
By program, e.g. task dispatcher, supervisor, operating system: scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
G06F 11/0709
The processing taking place on a specific hardware platform or in a specific software environment: in a distributed system consisting of a…
G06F 11/0751
Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of…