# US 2015/0106343 A1 · Technique for global deduplication across datacenters with minimal coordination

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0106343 A1
- Document ID: US20150106343A1
- Status: Published application
- Kind: A1
- Application number: 14/054,984
- Filed: 2013-10-16
- Published: 2015-04-16
- Earliest claimed: 2013-10-16
- Assignees: NetApp, Inc.
- Applicants: NetApp, Inc.
- Inventors: Kiran Nenmeli Srinivasan; Kishore Kasi Udayashankar; Swetha Krishnan
- Web page: https://patentgrep.com/patent/US20150106343A1

## CPC

- G06F 16/2365: {Ensuring data consistency and integrity}
- G06F 3/0608: {Saving storage space on storage systems}
- G06F 3/0641: {De-duplication techniques}
- G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]}
- G06F 16/2228: {Indexing structures}
- G06F 16/24568: {Data stream processing; Continuous queries}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system and method for global data de-duplication in a cloud storage environment utilizing a plurality of data centers is provided. Each cloud storage gateway appliance divides a data stream into a plurality of data objects and generates a content-based hash value as a key for each data object. An IMMUTABLE PUT operation is utilized to store the data object at the associated key within the cloud.

## Claims

### Claim 1

A method for de-duplicating a data stream, the method comprising:
chunking the data stream into a plurality of data objects;
calculating a key associated with each of the data objects;
performing an immutable put operation to store each data object in a cloud storage environment, each data object indexed by its associated calculated key; and
updating metadata associated with the data stream to enable subsequent retrieval of the data objects from the cloud storage environment, whereby communication among a plurality of data centers storing data in the cloud storage environment is obviated.

### Claim 2 (depends on 1)

The method of claim 1 wherein the chunking of the data stream into a plurality of data objects comprises performing a variable length chunking procedure to generate the plurality of data objects.

### Claim 3 (depends on 2)

The method of claim 2 wherein the variable length chunking procedure utilizes a two thresholds two divisors algorithm.

### Claim 4 (depends on 1)

The method of claim 1 wherein the key comprises a concatenation of block keys, wherein each block key is associated with a fixed sized data block of the data object.

### Claim 5 (depends on 1)

The method of claim 1 further comprising:
computing a content-based hash for each of a plurality of data blocks within a data object; and
computing the key by concatenating the plurality of calculated content-based hashes.

### Claim 6 (depends on 1)

The method of claim 1 wherein updating metadata further comprise updating a mappings data structure.

### Claim 7 (depends on 6)

The method of claim 6 wherein the mappings data structure stores the calculated keys in an order to enable reconstruction of the data stream.

### Claim 8 (depends on 6)

The method of claim 6 wherein the mappings data structure comprises a plurality of entries, each of the plurality of entries configured to store one of the plurality of calculated keys.

### Claim 9 (depends on 1)

The method of claim 1 wherein the data stream comprises a backup data stream.

### Claim 10 (depends on 1)

The method of claim 1 wherein the data stream comprises a virtual tape library data stream.

### Claim 11 (depends on 1)

The method of claim 1 wherein the cloud storage environment comprises a private cloud storage environment.

### Claim 12 (depends on 1)

The method of claim 1 wherein the cloud storage environment comprises a public cloud storage environment.

### Claim 13

A system for de-duplicating a data stream, the system comprising:
a cloud storage gateway appliance operatively interconnected with a cloud storage system via a network;
one or more clients operatively interconnected with the cloud storage gateway appliance, at least one of the one or more clients configured to forward the data stream to the cloud storage gateway appliance;
wherein the cloud storage gateway appliance is configured to chunk the data stream into a plurality of data objects, compute a key for each of the data objects and execute an immutable put operation to store each data object with its associated key in the cloud storage system.

### Claim 14 (depends on 7)

The system of claim 7 wherein the cloud storage gateway appliance comprises a storage system.

### Claim 15 (depends on 13)

The storage system of claim 13 wherein the cloud storage gateway appliance comprises a backup server.

### Claim 16 (depends on 13)

The storage system of claim 13 wherein the key comprises a concatenation of block keys, wherein each block key is associated with a fixed sized data block of the data object.

### Claim 17 (depends on 13)

The storage system of claim 13 wherein the cloud storage gateway appliance is configured to chunk the data streak into a plurality of data objects using a variable length chunking procedure to generate the plurality of data objects.

### Claim 18 (depends on 17)

The storage system of claim 17 wherein the variable length chunking procedure comprises two thresholds two divisors algorithm.

### Claim 19 (depends on 13)

The storage system of claim 13 wherein the cloud storage system comprises a public cloud storage system.

### Claim 20

A non-transitory computer readable medium, the non-transitory computer readable medium comprising:
program instructions for chunking the data stream into a plurality of data objects;
program instructions for calculating a key associated with each of the data objects;
program instructions for performing an immutable put operation to store each data object in a cloud storage environment, each data object indexed by its associated calculated key; and
program instructions for updating metadata associated with the data stream to enable subsequent retrieval of the data objects from the cloud storage environment.
