# US 2021/0279248 A1 · Live data conversion and migration for distributed data object systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279248 A1
- Document ID: US20210279248A1
- Status: Published application
- Kind: A1
- Application number: 16/878,134
- Filed: 2020-05-19
- Published: 2021-09-09
- Earliest claimed: 2020-03-06
- Assignees: Dropbox, Inc.
- Applicants: Dropbox, Inc.
- Inventors: Ashish Gandhi; Renjish Abraham; Kevin Farrell
- Web page: https://patentgrep.com/patent/US20210279248A1

## CPC

- G06F 16/258: {Data format conversion from or to a database}
- G06F 11/1004: {to protect a block of data words, e.g. CRC or checksum (G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00)}
- G06F 11/3034: {where the computing system component is a storage system, e.g. DASD based or network based (digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097)}
- G06F 16/214: {Database migration support}
- G06F 16/2282: {Tablespace storage structures; Management thereof}
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 16/284: {Relational databases}
- G06F 21/6218: {to a system of files or objects, e.g. local or distributed file system or database}
- G06F 21/64: Protecting data integrity, e.g. using checksums, certificates or signatures
- H04L 63/123: {received data contents, e.g. message integrity}
- H04L 63/1416: {Event detection, e.g. attack signature detection}
- H04L 63/1458: {Denial of Service}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Techniques are disclosed for converting the data of data objects stored in a DDOS from one data format to another while the system is live and available to process requests for access to the data objects being converted. The conversion techniques do not require taking the DDOS offline. The conversion techniques also do not require locking a data object for the entire conversion of the data object.

## Claims

### Claim 1

A computing system comprising:
a plurality of processors;
storage media; and
instructions stored in the storage media which, when executed by the plurality of processors, cause the computing system to perform:
storing a first set of database tables in a target data format based on a second set of database tables in a source data format;
wherein the first set of database tables and the second set of database tables are stored in a distributed data object system;
obtaining a first set of one or more signatures for a target database table of the first set of database tables in the target data format;
for a source database table of the second set of database tables in the source data format:
acquiring a lock on the source database table,
obtaining a second set of one or more signatures for the source database table,
comparing the first set of one or more signatures for the target database table to the second set of one or more signatures for the source database table,
based on the comparing the first set of one or more signatures to the second set of one or more signatures, determining a set of one or more mismatched database table rows between the target database table and the source database table, and
copying the set of one or more mismatched database table rows from the source database table in the source data format to the target database table in the target data format.

### Claim 2 (depends on 1)

The computing system of claim 1, further comprising:
instructions stored in the storage media which, when executed by the plurality of processors, cause the computing system to perform:
for the source database table in the second set of database tables:
determining that a number of mismatched rows in the set of one or more mismatched database table rows does not exceed a threshold number of mismatched rows, and
releasing the lock on the source database table after performing the copying the set of one or more mismatched database table rows from the source database table in the source data format to the target database table in the target data format.

### Claim 3 (depends on 1)

The computing system of claim 1, further comprising:
instructions stored in the storage media which, when executed by the plurality of processors, cause the computing system to perform:
for the source database table in the second set of database tables:
determining that a number of mismatched rows in the set of one or more mismatched database table rows exceeds a threshold number of mismatched rows, and
releasing the lock on the source database table before performing the copying the set of one or more mismatched database table rows from the source database table in the source data format to the target database table in the target data format.

### Claim 4 (depends on 1)

The computing system of claim 1, further comprising:
instructions stored in the storage media which, when executed by the plurality of processors, cause the computing system to perform:
obtaining the first set of one or more signatures from a metadata store of the distributed data object system.

### Claim 5 (depends on 1)

The computing system of claim 1, further comprising:
instructions stored in the storage media which, when executed by the plurality of processors, cause the computing system to perform:
computing the first set of one or more signatures from the target database table.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein the source data format is a record columnar format and the target data format is an optimized row columnar format.

### Claim 7

A computer-implemented method comprising:
storing a first set of data objects in a target data format based on a second set of data objects in a source data format;
wherein the first set of data objects and the second set of data objects are stored in a distributed data object system;
obtaining a first set of one or more signatures for a target data object of the first set of data objects in the target data format;
for a source data object of the second set of data objects in the source data format:
acquiring a lock on the source data object,
obtaining a second set of one or more signatures for the source data object,
comparing the first set of one or more signatures for the target data object to the second set of one or more signatures for the source data object,
based on the comparing the first set of one or more signatures to the second set of one or more signatures, determining a set of one or more mismatches between the target data object and the source data object, and
copying the set of one or more mismatches from the source data object in the source data format to the target data object le in the target data format.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, further comprising:
for the source data object in the second set of data objects:
releasing the lock on the source data object before performing the copying the set of one or more mismatches from the source data object in the source data format to the target data object in the target data format.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, further comprising:
for the source data object in the second set of data objects:
releasing the lock on the source data object after performing the copying the set of one or more mismatches from the source data object in the source data format to the target data object in the target data format

### Claim 10 (depends on 7)

The computer-implemented method of claim 7, wherein the first set of data objects and the second set of data objects each comprise at least one million data objects.

### Claim 11 (depends on 7)

The computer-implemented method of claim 7, wherein the second set of data objects in the source data format is writable during the storing the first set of data objects in the target data format based on the second set of data objects in the source data format.

### Claim 12 (depends on 7)

The computer-implemented method of claim 7, wherein the set of one or more mismatches between the source data object and the target data object is caused by one or more data objects in the second set of data objects being modified during the storing the first set of data objects in the target data format based on the second set of data objects in the source data format.

### Claim 13 (depends on 7)

The computer-implemented method of claim 7, wherein the source data format is a record columnar format and the target data format is an optimized row columnar format.

### Claim 14 (depends on 7)

The computer-implemented method of claim 7, wherein the lock acquired on the source data object is a write lock that allows the distributed data object system to read the source data object.

### Claim 15 (depends on 7)

The computer-implemented method of claim 7, wherein:
the set of one or more mismatches corresponds to a set of one or more database table rows of the source data object;
the copying the set of one or more mismatches from the source data object in the source data format to the target data object in the target data format comprises copying the set of one or more database table rows to the target data object; and
the set of one or more database tables rows copied to the target data object are stored in the target data object according to the target data format.

### Claim 16 (depends on 7)

The computer-implemented method of claim 7, further comprising:
using a set of long-lived worker processes to perform the storing the first set of data objects in the target data format based on the second set of data objects in a source data format.

### Claim 17

One or more non-transitory storage media storing instructions which, when executed by a computing system having one or more processors, cause the computing system to perform:
creating a first set of database tables in a target data format from a second set of database tables in a source data format;
wherein the first set of database tables and the second set of database tables are stored in a distributed data object system;
obtaining a first set of signatures for a target database table of the first set of database tables in the target data format;
for a source database table of the second set of database tables in the source data format:
acquiring a lock on the source database table,
obtaining a second set of signatures for the source database table,
comparing the first set of signatures for the target database table to the second set of signatures for the source database table,
based on the comparing the first set of signatures to the second set of signatures, determining a set of mismatched database table rows between the target database table and the source database table, and
copying the set of mismatched database table rows from the source database table in the source data format to the target database table in the target data format.

### Claim 18 (depends on 17)

The one or more non-transitory storage media of claim 17, further comprising:
instructions which, when executed by the computing system having one or more processors, cause the computing system to perform:
for the source database table in the second set of database tables:
releasing the lock on the source database table before performing the copying the set of mismatched database table rows from the source database table in the source data format to the target database table in the target data format.

### Claim 19 (depends on 17)

The one or more non-transitory storage media of claim 17, further comprising:
instructions which, when executed by the computing system having one or more processors, cause the computing system to perform:
for the source database table in the second set of database tables:
releasing the lock on the source database table after performing the copying the set of mismatched database table rows from the source database table in the source data format to the target database table in the target data format.

### Claim 20 (depends on 17)

The one or more non-transitory storage media of claim 17, wherein the source data format is a record columnar format and the target data format is an optimized row columnar format.
