US 8,209,334 B1 · Method to direct data to a specific one of several repositories

Abstract

Data de-duplication schemes reduce the amount of storage necessary to store a data set by dividing the data into segments and storing a segment identifier on a storage medium in place of each data segment. Each unique data segment is stored in a repository, and duplicate data segments are not stored. Methods and apparatus are provided for distributing data segments across multiple repositories in a data storage system, thereby reducing the quantity of data stored at a particular repository. Segments are assigned to repositories based upon a characteristic of the segments. The characteristic may be the length of the segment or some other value produced by a repeatable, uniformly-distributed function of the segment. The characteristic may be stored on the storage medium along with the segment identifier. The original data may be regenerated by retrieving the segment identifiers and characteristics from the storage medium and retrieving each segment from the repository identified by the characteristic.

Claims14

Dependent claims

Segment distribution logic for distributing a segment of data across multiple repositories located on multiple apparatuses in a data storage system and for distributing an index associated with the data storage system across the multiple repositories, the logic comprising:

segmentation logic for dividing data into at least one segment according to a segmentation scheme;
characteristic generation logic for generating a characteristic based upon the at least one segment, where the characteristic generation logic is operable to repeatedly produce the same characteristic for the same segment, and the characteristic generation logic is further operable to generate a substantially uniform distribution of characteristics for different segments;
repository selection logic for generating a repository identifier based upon the characteristic;
communication logic for communicating the segment and a segment identifier that refers to the segment to a repository identified by the repository identifier, the repository being located on a remote computer, and
output logic for storing on a storage medium the segment and the segment identifier, where the segment identifier comprises the repository identifier and a key based upon the segment of data, where the output logic is operable to generate the key using a hash function, and where the storage medium is configured to store the segment and an index for locating the segment,
where the characteristic comprises a cryptographic hash value computed from the segment, a simpler hash value based on the segment, or a length of the segment, and
where the repository selection logic is operable to select the repository identifier, based upon the characteristic, from a mapping table.

4 more elements

A computer system, comprising:

a processor;
a memory;
a set of logics;
and
an interface to connect the processor, the memory, and the set of logics, the set of logics comprising:
segment distribution logic configured to distribute different data segments to multiple different repositories located on multiple different apparatuses, the segment distribution logic comprising:
segmentation logic configured to divide data into two or more different data segments according to a segmentation scheme;
characteristic generation logic configured to generate a characteristic for a data segment as a function of the contents of the data segment, the function being repeatable and deterministic, the function being configured to produce a substantially uniform distribution of characteristics for the two or more different data segments;
repository selection logic for generating a repository identifier for a data segment based upon the characteristic of the data segment, where the repository selection logic is configured to produce different repository identifiers for different data segments having different characteristics;
and
communication logic configured to communicate a data segment and a characteristic to a repository selected as a function of the repository identifier associated with the data segment, the repository being one of multiple repositories located on multiple apparatuses, the repository being located on a remote apparatus;
segment re-assembly logic configured to reassemble original data from data segments distributed across multiple different repositories located on multiple remote apparatuses, the segment re-assembly logic comprising:
segment identifier retrieval logic configured to retrieve at least one characteristic from a storage medium;
repository selection logic configured to generate a repository identifier for a data segment based upon the characteristic of the data segment, where the repository selection logic is configured to produce different repository identifiers for different data segments having different characteristics;
and
communication logic configured to receive a data segment from a repository selected as a function of the repository identifier, the repository being located on a remote apparatus
where the characteristic comprises a cryptographic hash value computed from the segment, a non-cryptographic hash value based on the segment, or a length of the segment, and
where generating the repository identifier comprises selecting the repository identifier, based upon the characteristic, from a mapping table.

15 more elements

Dependencies are inferred from the claim text.

Method to direct data to a specific one of several repositories

Life of this patent

Method to direct data to a specific one of several repositories

Life of this patent

Abstract

Claims14