Punctuation controlled temporal reference data lookup
AssigneeMicrosoft Technology Licensing, LLC
InventorsBoris Shulman, Shoupei Li, Alexander Alperovich, Xindi Zhang, Kanstantsyn Zoryn
Life of this patent
Estimating the term…
Abstract
A method for joining an event stream with reference data includes loading a plurality of reference data snapshots from a reference data source into a cache. Punctuation events are supplied that indicate temporal validity for the plurality of reference data snapshots in the cache. A logical barrier is provided that restricts a flow of data events in the event stream to a cache lookup operation based on the punctuation events. The cache lookup operation is performed with respect to the data events in the event stream that are permitted to cross the logical barrier.
Claims18
A method for joining an event stream with reference data, the method being implemented by a compute processor in an event stream processing system, the method comprising:
- sending a request to a reference data manager;
- receiving a punctuation event from the reference data manager in response to the request, wherein the punctuation event comprises a punctuation event timestamp, wherein the reference data manager determines the punctuation event timestamp by selecting a highest timestamp out of a plurality of timestamps associated with a plurality of reference data snapshots in a cache, and wherein the reference data manager causes at least one new reference data snapshot to be added to the cache when a difference between the highest timestamp out of the plurality of timestamps associated with the plurality of reference data snapshots in the cache and a timestamp that is associated with a most recent data request sent by the compute processor is less than a predefined threshold value;
- providing a logical barrier that restricts a flow of data events in the event stream to a cache lookup operation based on the punctuation event;
- performing a first cache lookup operation with respect to a first data event in the event stream, the first data event having a first timestamp that is earlier than the punctuation event timestamp;
- and
- delaying a second cache lookup operation from being performed with respect to a second data event in the event stream, the second data event having a second timestamp that is later than the punctuation event timestamp.
A computer-readable medium comprising instructions that are executable by a compute processor in an event processing system to:
- send a request to a reference data manager;
- receive a punctuation event from the reference data manager in response to the request, wherein the punctuation event comprises a punctuation event timestamp, wherein the reference data manager determines the punctuation event timestamp by selecting a highest timestamp out of a plurality of timestamps associated with a plurality of reference data snapshots in a cache, and wherein the reference data manager causes at least one new reference data snapshot to be added to the cache when a difference between the highest timestamp out of the plurality of timestamps associated with the plurality of reference data snapshots in the cache and a timestamp that is associated with a most recent data request sent by the compute processor is less than a predefined threshold value;
- provide a logical barrier that restricts a flow of data events in an event stream to a cache lookup operation based on the punctuation event;
- perform a first cache lookup operation with respect to a first data event in the event stream, the first data event having a first timestamp that is earlier than the punctuation event timestamp;
- and
- delay a second cache lookup operation from being performed with respect to a second data event in the event stream, the second data event having a second timestamp that is later than the punctuation event timestamp.
An event processing system that is configured to join an event stream with reference data, comprising:
- one or more processors;
- memory in electronic communication with the one or more processors;
- and
- instructions stored in the memory, the instructions being executable by the one or more processors to:
- send a request to a reference data manager;
- receive a punctuation event from the reference data manager in response to the request, wherein the punctuation event comprises a punctuation event timestamp, wherein the reference data manager determines the punctuation event timestamp by selecting a highest timestamp out of a plurality of timestamps associated with a plurality of reference data snapshots in a cache, and wherein the reference data manager causes at least one new reference data snapshot to be added to the cache when a difference between the highest timestamp out of the plurality of timestamps associated with the plurality of reference data snapshots in the cache and a timestamp that is associated with a most recent data request sent by the compute processor is less than a predefined threshold value;
- provide a logical barrier that restricts a flow of data events in the event stream to a cache lookup operation based on the punctuation event;
- perform a first cache lookup operation with respect to a first data event in the event stream, the first data event having a timestamp that is earlier than the punctuation event timestamp;
- and
- delay a second cache lookup operation from being performed with respect to a second data event in the event stream, the second data event having a second timestamp that is later than the punctuation event timestamp.
Dependencies are inferred from the claim text.