US 2022/0035829 A1 · TECHNIQUES AND ARCHITECTURES FOR MANAGING CASCADING MUTATIONS IN AN ENVIRONMENT HAVING A DATA LAKE

Abstract

Managing mutations in a data lake environment. A mutation request to cause write operations that modify data objects or structures within an environment for collecting unformatted raw data is received. The environment has at least a data table and a notification table. An entry is written to the data table with a streaming job configured to receive and process the mutation request. Entries to the data table specify at least records indicating changes to objects in the environment based on ingestion processing for the environment for collecting unformatted raw data and based on the mutation request. A corresponding entry is written to the notification table in response to a successful write attempt to the data table. The notification table entry has information about data table entries for a specified period. At least one data consumer is notified that the data table has been modified.

Claims18

Dependent claims

A method for managing mutations of data records, the method comprising:

receiving a mutation request to cause one or more write operations that modify data objects or structures within an environment for collecting unformatted raw data, the environment having at least a data table and a notification table;
attempting to write an entry to the data table with a streaming job configured to receive and process the mutation request, wherein entries to the data table specify at least records indicating changes to objects in the environment based on ingestion processing for the environment for collecting unformatted raw data and based on the mutation request;
attempting to write a corresponding entry to the notification table in response to a successful write attempt to the data table, wherein the notification table entry comprises information about data table entries for a specified period;
notifying at least one data consumer that the data table has been modified utilizing the corresponding entry to the notification table.

1 more element

A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, are configurable to cause the one or more processors to:

receive a mutation request to cause one or more write operations that modify data objects or structures within an environment for collecting unformatted raw data, the environment having at least a data table and a notification table;
attempt to write an entry to the data table with a streaming job configured to receive and process the mutation request, wherein entries to the data table specify at least records indicating changes to objects in the environment based on ingestion processing for the environment for collecting unformatted raw data and based on the mutation request;
attempt to write a corresponding entry to the notification table in response to a successful write attempt to the data table, wherein the notification table entry comprises information about data table entries for a specified period;
notify at least one data consumer that the data table has been modified utilizing the corresponding entry to the notification table.

1 more element

A system comprising:

a physical storage system;
one or more hardware processors coupled with the storage system, the one or more hardware processors to receive a mutation request to cause one or more write operations that modify data objects or structures within an environment for collecting unformatted raw data, the environment having at least a data table and a notification table, to attempt to write an entry to the data table with a streaming job configured to receive and process the mutation request, wherein entries to the data table specify at least records indicating changes to objects in the environment based on ingestion processing for the environment for collecting unformatted raw data and based on the mutation request, to attempt to write a corresponding entry to the notification table in response to a successful write attempt to the data table, wherein the notification table entry comprises information about data table entries for a specified period, and to notify at least one data consumer that the data table has been modified utilizing the corresponding entry to the notification table.

Dependencies are inferred from the claim text.

Techniques and architectures for managing cascading mutations in an environment having a data lake

Life of this patent

Techniques and architectures for managing cascading mutations in an environment having a data lake

Life of this patent

Abstract

Claims18