Ensuring exactly once data ingestion
AssigneeFortinet, Inc.
InventorsJean-Philippe E. Martin, Úlfar Erlingsson, Yijou Chen
Life of this patent
Estimating the term…
Abstract
Ensuring exactly once data ingestion, the method comprising: receiving, from an event stream, event data comprising a plurality of first records; generating a plurality of second records each corresponding to a first record of the plurality of first records and comprising a log sequence number (LSN) describing a location in the event stream for a corresponding first record; and sending, to a data warehouse, the plurality of second records.
Claims18
A method of ensuring exactly once data ingestion, the method comprising:
- interposing an event streaming platform and a data ingestion pipeline between a computing environment and a data warehouse platform;
- receiving, by the data ingestion pipeline, from an event stream of the event streaming platform, event data comprising a plurality of first records, wherein the event data is received in real-time as it is collected from respective compute assets of a plurality of compute assets operable in the computing environment;
- generating, by the data ingestion pipeline, a plurality of second records each corresponding to a first record of the plurality of first records and comprising a log sequence number (LSN) describing a location in the event stream for a corresponding first record;
- sending, by the data ingestion pipeline, to a data warehouse, the plurality of second records;
- and
- after detecting occurrence of an error or a restart of the data ingestion pipeline, ensuring a given second record of the plurality of second records is stored exactly once in the data warehouse by:
- querying, by the data ingestion pipeline, the data warehouse for an LSN for a last stored second record;
- and
- requesting, by the data ingestion pipeline, from the event streaming platform, next event data based on the LSN for the last stored second record.
A computer program product for ensuring exactly once data ingestion, the computer program product embodied on a non-transitory computer readable medium, the computer program product including computer program instructions configurable to carry out the steps of;
- receiving, by a data ingestion pipeline interposed between a computing environment and a data warehouse platform, from an event stream of an event streaming platform interposed between the data ingestion pipeline and the computing environment, event data comprising a plurality of first records;
- generating, by the data ingestion pipeline, a plurality of second records each corresponding to a first record of the plurality of first records and comprising a log sequence number (LSN) describing a location in the event stream for a corresponding first record;
- and
- sending, to a data warehouse, the plurality of second records;
- and
- after detecting occurrence of an error or a restart of the data ingestion pipeline, ensuring a given second record of the plurality of second records is stored exactly once in the data warehouse by:
- querying, by the data ingestion pipeline, the data warehouse for an LSN for a last stored second record;
- and
- requesting, by the data ingestion pipeline, from the event streaming platform, next event data based on the LSN for the last stored second record.
Dependencies are inferred from the claim text.