# US 2024/0004898 A1 · Updating a staged dataset for ingestion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004898 A1
- Document ID: US20240004898A1
- Status: Published application
- Kind: A1
- Application number: 18/057,516
- Filed: 2022-11-21
- Published: 2024-01-04
- Earliest claimed: 2022-07-01
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Vineet BINDAL; Manohar SREENIVASACHAR; Michael Allan JOHNSTON; Lengning LIU
- Web page: https://patentgrep.com/patent/US20240004898A1

## CPC

- G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses}
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 11/1451: {by selection of backup contents}
- G06F 16/219: {Managing data history or versioning (querying versioned data G06F16/2474; querying temporal data G06F16/2477)}
- G06F 16/258: {Data format conversion from or to a database}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Updating a staged dataset from a delta-based data repository is provided, including receiving an instruction to update the staged dataset based on a checkpoint and requesting a versioned subset of data from the repository based on the checkpoint and the received instruction. The versioned subset of the data includes a data differential between a staged version of the data stored in the repository and a checkpoint-based version of the data stored in the repository. The versioned subset of the data is received from the repository. The versioned subset of the data is incompatibly formatted for ingestion by a data analytics engine. The received versioned subset of the data is transformed into a staged subset of the data. The staged subset of the data is formatted for ingestion by a data analytics engine and stored in association with another staged subset of the data for ingestion by the data analytics engine.

## Claims

### Claim 1

A method of updating a staged dataset from a delta-based data repository, the method comprising:
receiving an instruction to update the staged dataset based on a checkpoint;
requesting a versioned subset of data from the delta-based data repository based on the checkpoint and the received instruction, wherein the versioned subset of the data includes a data differential between a staged version of the data stored in the delta-based data repository and a checkpoint-based version of the data stored in the delta-based data repository;
receiving the versioned subset of the data from the delta-based data repository, wherein the versioned subset of the data is incompatibly formatted for ingestion by a data analytics engine;
transforming the received versioned subset of the data into a staged subset of the data, wherein the staged subset of the data is formatted for ingestion by a data analytics engine; and
storing the staged subset of the data in association with another staged subset of the data for ingestion by the data analytics engine.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
communicating the staged subset of the data to the data analytics engine for ingestion.

### Claim 3 (depends on 1)

The method of claim 1, wherein the checkpoint is communicated with the received instruction.

### Claim 4 (depends on 1)

The method of claim 1, wherein the instruction is received by a request generator of a data transformer and the checkpoint is obtained from storage by the request generator of the data transformer.

### Claim 5 (depends on 1)

The method of claim 1, wherein the transforming operation reformats the data received from the delta-based data repository into a format that is compatible for ingestion by the data analytics engine.

### Claim 6 (depends on 1)

The method of claim 1, wherein the delta-based data repository includes a delta lake.

### Claim 7 (depends on 1)

The method of claim 1, wherein the delta-based data repository includes a data lake.

### Claim 8

A system for updating a staged dataset from a delta-based data repository, the system comprising:
one or more hardware processors;
an input interface of a data transformer, the input interface being executable by the one or more hardware processors and configured to receive an instruction to update the staged dataset based on a checkpoint;
a request generator of the data transformer executable by the one or more hardware processors and configured to request a versioned subset of data from the delta-based data repository based on the checkpoint and the received instruction, wherein the versioned subset of the data includes a data differential between a staged version of the data stored in the delta-based data repository and a checkpoint-based version of the data stored in the delta-based data repository, wherein the input interface is further configured to receive the versioned subset of the data from the delta-based data repository, wherein the versioned subset of the data is incompatibly formatted for ingestion by a data analytics engine;
an ingestion staging engine of a data transformer executable by the one or more hardware processors and configured to transform the received versioned subset of the data into a staged subset of the data, wherein the staged subset of the data is formatted for ingestion by a data analytics engine; and
an output interface of the data transformer executable by the one or more hardware processors and configured to store the staged subset of the data in association with another staged subset of the data for ingestion by the data analytics engine.

### Claim 9 (depends on 8)

The system of claim 8, further comprising:
a data ingestor executable by the one or more hardware processors and configured to communicate the staged subset of the data to the data analytics engine for ingestion.

### Claim 10 (depends on 8)

The system of claim 8, wherein the checkpoint is communicated with the received instruction.

### Claim 11 (depends on 8)

The system of claim 8, wherein the instruction is received by the request generator of a data transformer and the checkpoint is obtained from storage by the request generator of the data transformer.

### Claim 12 (depends on 8)

The system of claim 8, wherein the ingestion staging engine reformats the data received from the delta-based data repository into a format that is compatible for ingestion by the data analytics engine.

### Claim 13 (depends on 8)

The system of claim 8, wherein the delta-based data repository includes a delta lake.

### Claim 14 (depends on 8)

The system of claim 8, wherein the delta-based data repository includes a data lake.

### Claim 15

One or more tangible processor-readable storage media embodied with instructions for executing on one or more processors and circuits of a computing device a process for updating a staged dataset from a delta-based data repository, the process comprising:
receiving an instruction to update the staged dataset based on a checkpoint;
requesting a versioned subset of data from the delta-based data repository based on the checkpoint and the received instruction, wherein the versioned subset of the data includes a data differential between a staged version of the data stored in the delta-based data repository and a checkpoint-based version of the data stored in the delta-based data repository;
receiving the versioned subset of the data from the delta-based data repository, wherein the versioned subset of the data is incompatibly formatted for ingestion by a data analytics engine;
transforming the received versioned subset of the data into a staged subset of the data, wherein the staged subset of the data is formatted for ingestion by a data analytics engine; and
storing the staged subset of the data in association with another staged subset of the data in a staged dataset for ingestion by the data analytics engine.

### Claim 16 (depends on 15)

The one or more tangible processor-readable storage media of claim 15, wherein the process further comprises:
communicating the staged subset of the data to the data analytics engine for ingestion.

### Claim 17 (depends on 15)

The one or more tangible processor-readable storage media of claim 15, wherein the checkpoint is communicated with the received instruction.

### Claim 18 (depends on 15)

The one or more tangible processor-readable storage media of claim 15, wherein the instruction is received by a request generator of a data transformer and the checkpoint is obtained from storage by the request generator of the data transformer.

### Claim 19 (depends on 15)

The one or more tangible processor-readable storage media of claim 15, wherein the transforming operation reformats the data received from the delta-based data repository into a format that is compatible for ingestion by the data analytics engine.

### Claim 20 (depends on 15)

The one or more tangible processor-readable storage media of claim 15, wherein the delta-based data repository includes a delta lake.
