US 8,478,951 B1Granted

Method and apparatus for block level data de-duplication

AssigneeEMC Corporation

InventorsMichael W. Healey, J. Michael Dunbar, Avinash Kallat, Michael Craig Fishman

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques for performing de-duplication for data blocks in a computer storage environment. At least one chunking/hashing unit receives input data from a source and processes it to output data blocks and content addresses for them. In one aspect, the chunking/hashing unit outputs all blocks without checking to see whether any is a duplicate of a block previously stored on the storage environment. In another aspect, each data block is processed by one of a plurality of distributed object addressable storage (OAS) devices that each is selected to process data blocks having content addresses with a particular range. The OAS devices determine whether each received to data block is a duplicate of another previously stored on the computer storage environment, and when it is not, stores the data block.

Claims16

Dependent claims

A computer storage environment comprising:

at least one chunking/hashing unit that receives input data from at least one source, wherein the at least one chunking/hashing unit processes at least some of the input data to output a plurality of data blocks from the at least some of the input data and a content address for each of the plurality of data blocks, wherein a content address for a corresponding data block is generated based, at least in part, on the content of the corresponding data block;
and
a plurality of object addressable storage devices to store at least some of the plurality of data blocks output from the at least one chunking/hashing unit;
wherein the computer storage environment comprises at least one processor programmed to, for each one of the plurality of data blocks output from the at least one chunking/hashing unit, make a determination as to which of the plurality of object addressable storage devices is to control storage of the one of the plurality of data blocks output from the at least one chunking/hashing unit;
and
wherein each of the plurality of object addressable storage devices comprises at least one processor programmed to, in response to receipt from the at least one chunking/hashing unit of a received one of the plurality of data blocks, process the received one of the plurality of data blocks to:
determine whether the received one of the plurality of data blocks is a duplicate of another data block previously stored on the computer storage environment;
control storage of the received one of the plurality of data blocks on the computer storage environment when it is determined that the received one of the plurality of data blocks is not a duplicate of another data block previously stored on the computer storage environment, wherein the received one of the plurality of data blocks is stored such that it is accessible to the computer storage environment using the corresponding content address;
and
control storage of information indicating that the received one of the plurality of data blocks is represented by data previously stored on the computer storage environment when it is determined that the received one of the plurality of data blocks is a duplicate of another data block previously stored on the computer storage environment,
wherein two or more of the plurality of object addressable storage devices process a respective received one of the plurality of data blocks in parallel.

8 more elements

A method comprising acts of:

(A) processing at least some received input data at a hashing/chunking unit to output a plurality of data blocks from the at least some of the input data and a content address for each of the plurality of data blocks, wherein a content address for a corresponding data block is generated based, at least in part, on the content of the corresponding data block;
(B) processing, in parallel, at least some of the plurality of data blocks at two or more of a plurality of object addressable storage devices, the two or more of the plurality of object addressable storage devices determined from among the plurality of object addressable storage devices based upon the content address of the respective data blocks, to determine whether the at least some of the plurality of data blocks are duplicates of another data block previously stored on the plurality of object addressable storage devices;
and
(C) storing on at least one of the plurality of object addressable storage devices each one of the plurality of data blocks determined in the act (B) to not be a duplicate of another data block previously stored on the plurality of object addressable storage devices without storing a data block of the plurality of data blocks that is determined in the act (B) to be a duplicate of another data block previously stored on the plurality of object addressable storage devices, wherein storing each one of the plurality of data blocks determined in act (B) to not be a duplicate of another data block is based, at least in part, on each respective content address.

1 more element

Dependencies are inferred from the claim text.

Facts

Application: 13/447,048Filed 13 Apr 2012 · continuation
Priority: 31 Dec 2008
Granted: 2 Jul 2013
Term ends: Not estimatedThe source records don’t support an estimate.

Classification6 codes

G06F 3/0641
Management of blocks: de-duplication techniques
G06F 11/1453
Management of the data involved in backup or backup restore: using de-duplication of the data
G06F 3/0608
Specifically adapted to achieve a particular effect: saving storage space on storage systems
G06F 3/0683
In-line storage system: plurality of storage devices