US 2023/0055677 A1 · SYSTEMS AND METHODS FOR DATA LINKAGE AND ENTITY RESOLUTION OF CONTINUOUS AND UN-SYNCHRONIZED DATA STREAMS

Abstract

The present disclosure is directed to a scalable, extensible, fault-tolerant system for stateful joining of two or more streams that are not fully synchronized, event ordering is not guaranteed, and certain events arrive a bit late. The system can ensure to combine the events or link the data in near real-time with low latency to mitigate impacts on downstream applications, such as ML models for determining suspicious behavior. Apart from combining events, the system can ensure to propagate the needed entities to other product streams or help in entity resolution. If any of the needed data is yet to arrive, a user can configure a few parameters to achieve desired eventual and attribute consistency. The architecture is designed to be agnostic of stream processing framework and can work well with both streaming and batch paths.

Claims20

Dependent claims

A method comprising:

creating, by one or more processors, responsive to processing a first event from a first stream of a plurality of streams, an object including first data from the first event for merging the first data with second data from a second event of a second stream of the plurality of streams;
routing, by the one or more processors, the first event to a retry stream to reprocess the first event responsive to determining that the object does not include the second data from the second event and a number of times the first event has been routed to the retry stream does not satisfy a threshold;
and
transmitting, by the one or more processors, i) the first data and the second data included in the object responsive to processing the second event to update the object to include the second data or ii) the first data included in the object responsive to determining that the object does not include the second data from the second event and the number of times the first event has been routed to the retry stream satisfies the threshold.

1 more element

A system comprising:

one or more processors configured to:
create, responsive to receiving a first event from a first stream of a plurality of streams, an object including first data from the first event for merging the first data with second data from a second event of a second stream of the plurality of streams;
route the first event to a retry stream to reprocess the first event responsive to determining that the object does not include the second data from the second event and a number of times the first event has been routed to the retry stream does not satisfy a threshold;
and
transmit i) the first data and the second data included in the object responsive to processing the second event to update the object to include the second data or ii) the first data included in the object responsive to determining that the object does not include the second data from the second event and the number of times the first event has been routed to the retry stream satisfies the threshold.

2 more elements

Dependencies are inferred from the claim text.

Systems and methods for data linkage and entity resolution of continuous and un-synchronized data streams

Life of this patent

Systems and methods for data linkage and entity resolution of continuous and un-synchronized data streams

Life of this patent

Abstract

Claims20