# US 2022/0276993 A1 · Watermark-based techniques for change-data-capture

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0276993 A1
- Document ID: US20220276993A1
- Status: Published application
- Kind: A1
- Application number: 17/745,739
- Filed: 2022-05-16
- Published: 2022-09-01
- Earliest claimed: 2019-12-16
- Assignees: Netflix, Inc.
- Applicants: Netflix, Inc.
- Inventors: Andreas ANDREAKIS; Ioannis PAPAPANAGIOTOU
- Web page: https://patentgrep.com/patent/US20220276993A1

## CPC

- G06F 16/2358: {Change logging, detection, and notification (replication G06F16/27)}
- G06F 16/219: {Managing data history or versioning (querying versioned data G06F16/2474; querying temporal data G06F16/2477)}
- G06F 16/2308: {Concurrency control (transaction processing G06F9/466)}
- G06F 16/2365: {Ensuring data consistency and integrity}
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Various embodiments set forth systems and techniques for concurrent log and dump processing. The techniques include selecting, from a datastore, a chunk comprising one or more rows of data; comparing the one or more rows of data in the chunk with a first set of log events in a change log associated with the datastore, wherein each log event included in the first set of log events occurs after a first log event in the change log and prior to a second log event in the change log; selecting, based on the comparison, one or more non-conflicting rows in the chunk; and transmitting, to an output, one or more log events associated with the one or more non-conflicting rows in the chunk prior to processing a second set of log events in the change log, wherein the second set of log events occur after the second log event.

## Claims

### Claim 1

A computer-implemented method comprising the steps of:
selecting, from a datastore, a chunk comprising a plurality of rows of data;
comparing the plurality of rows of data in the chunk with a first plurality of log events that are associated with the chunk and included in a change log associated with the datastore;
determining that a first row of data included in the chunk conflicts with the first plurality of log events;
removing the first row of data from the chunk; and
transmitting, to an output, one or more log events associated with one or more rows of data remaining in the chunk.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more log events are transmitted to the output prior to processing a second plurality of log events included in the change log, wherein the second plurality of log events occurs after the first plurality of log events.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein each log event included in the first plurality of log events occurs after a first log event included in the change log and prior to a second log event included in the change log.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein a first log event included in the change log is associated with generating a low watermark in a watermark table, and wherein a second log event included in the change log is associated with generating a high watermark in the watermark table.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein a first log event included in the change log is associated with updating a row in a watermark table to reflect a specific universally unique identifier (UUID) value.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the chunk is selected from the datastore after a first log event included in the change log and prior to a second log event included in the change log.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising appending, to an output buffer associated with the output, the one or more log events associated with one or more rows of data remaining in the chunk.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the chunk is selected in response to a pause in processing the change log.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the plurality of rows of data in the chunk are compared with the first plurality of log events in response to a resumption in processing the change log.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein comparing the plurality of rows in the chunk with the first plurality of log events comprises comparing one or more primary keys associated with the plurality of rows in the chunk to one or more primary keys associated with plurality of log events.

### Claim 11

One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
selecting, from a datastore, a chunk comprising a plurality of rows of data;
comparing the plurality of rows of data in the chunk with a first plurality of log events that are associated with the chunk and included in a change log associated with the datastore;
determining that a first row of data included in the chunk conflicts with the first plurality of log events;
removing the first row of data from the chunk; and
transmitting, to an output, one or more log events associated with one or more rows of data remaining in the chunk.

### Claim 12 (depends on 11)

The one or more non-transitory computer readable media of claim 11, wherein the one or more log events are transmitted to the output prior to processing a second plurality of log events included in the change log, wherein the second plurality of log events occurs after the first plurality of log events.

### Claim 13 (depends on 11)

The one or more non-transitory computer readable media of claim 11, wherein each log event included in the first plurality of log events occurs after a first log event included in the change log and prior to a second log event included in the change log.

### Claim 14 (depends on 11)

The one or more non-transitory computer readable media of claim 11, wherein a first log event included in the change log is associated with generating a low watermark in a watermark table, and wherein a second log event included in the change log is associated with generating a high watermark in the watermark table.

### Claim 15 (depends on 11)

The one or more non-transitory computer readable media of claim 11, wherein a first log event included in the change log is associated with updating a row in a watermark table to reflect a specific universally unique identifier (UUID) value.

### Claim 16 (depends on 11)

The one or more non-transitory computer readable media of claim 11, wherein the chunk is selected from the datastore after a first log event included in the change log and prior to a second log event included in the change log.

### Claim 17 (depends on 11)

The one or more non-transitory computer readable media of claim 11, further comprising:
determining that a first log event in the change log comprising a low watermark event has been received; and
in response, comparing the plurality of rows of data in the chunk with the first plurality of log events.

### Claim 18 (depends on 11)

The one or more non-transitory computer readable media of claim 11, further comprising:
determining that a log event in the change log comprising a high watermark event has been received; and
in response, transmitting the one or more log events associated with one or more rows of data remaining in the chunk to the output.

### Claim 19 (depends on 11)

The one or more non-transitory computer readable media of claim 11, wherein the one or more log events associated with the one or more rows of data remaining in the chunk are serialized in a same format as the first plurality of log events.

### Claim 20

A system, comprising:
one or more processors; and
one or more memories coupled to the one or more processors and including one or more instructions that, when executed by the one or more processors, cause the one or more processors to:
select, from a datastore, a chunk comprising a plurality of rows of data;
compare the plurality of rows of data in the chunk with a first plurality of log events that are associated with the chunk and included in a change log associated with the datastore;
determine that a first row of data included in the chunk conflicts with the first plurality of log events;
remove the first row of data from the chunk; and
transmit, to an output, one or more log events associated with one or more rows of data remaining in the chunk.
