Capturing data lake changes
Assigneesalesforce.com, inc.
InventorsShreedhar SUNDARAM, Mahalaxmi SANATHKUMAR, Aaron ZHANG, Parin KENIA, Violet GONG, Priya SETHURAMAN, Zhidong KE, Kevin TERUSAK
Life of this patent
Estimating the term…
Abstract
A data lake partition identifier may be retrieved from a data lake update service. The data lake partition identifier may identify a partition of a data lake that stores a data lake records. Records may be retrieved using a query that includes one of the identifiers. Retrieved records may be transformed and transmitted to a downstream data service.
Claims20
A computer-implemented method comprising:
- retrieving from a data lake update service one or more data lake partition identifiers at a computing device having a processor and memory, each data lake partition identifier identifying a respective partition of a data lake, each of the identified partitions storing a respective plurality of data lake records, each plurality of data lake records including a respective subset of records having been updated within a designated period of time prior to a current time;
- retrieving one or more of the subset of records from the data lake partitions via a communication interface using one or more queries that each include a respective one or more of the data lake partition identifiers;
- generating a transformed one or more records by applying a transformation function to the retrieved one or more records;
- and
- transmitting the transformed one or more records to a downstream data service, the transformed one or more records being associated with the designated period of time.
A computing system implemented in a cloud computing environment, the computing system configured to perform a method comprising:
- retrieving from a data lake update service one or more data lake partition identifiers at a computing device having a processor and memory, each data lake partition identifier identifying a respective partition of a data lake, each of the identified partitions storing a respective plurality of data lake records, each plurality of data lake records including a respective subset of records having been updated within a designated period of time prior to a current time;
- retrieving one or more of the subset of records from the data lake partitions via a communication interface using one or more queries that each include a respective one or more of the data lake partition identifiers;
- generating a transformed one or more records by applying a transformation function to the retrieved one or more records;
- and
- transmitting the transformed one or more records to a downstream data service, the transformed one or more records being associated with the designated period of time.
One or more non-transitory machine-readable media having instructions stored thereon for performing a method, the method comprising:
- retrieving from a data lake update service one or more data lake partition identifiers at a computing device having a processor and memory, each data lake partition identifier identifying a respective partition of a data lake, each of the identified partitions storing a respective plurality of data lake records, each plurality of data lake records including a respective subset of records having been updated within a designated period of time prior to a current time;
- retrieving one or more of the subset of records from the data lake partitions via a communication interface using one or more queries that each include a respective one or more of the data lake partition identifiers;
- generating a transformed one or more records by applying a transformation function to the retrieved one or more records;
- and
- transmitting the transformed one or more records to a downstream data service, the transformed one or more records being associated with the designated period of time.
Dependencies are inferred from the claim text.