Back to results
US 8,447,740 B1Granted
Stream locality delta compression
AssigneeEMC Corporation
InventorsMark Huang, Philip Shilane, Grant Wallace, Nitin Garg, Edward K. Lee, Ming Benjamin Zhu, Kai Li
Life of this patent
Estimating the term…
Abstract
Stream locality delta compression is disclosed. A previous stream indicated locale of data segments is selected. A first data segment is then determined to be similar to a data segment in the stream indicated locale.
Claims71
A system for processing data, comprising:
- a stream locality system having a processor to select a previously stored stream indicated locale, the locale including a selection of a plurality of previously stored data segments satisfying a locality criterion with respect to a portion of a stream currently being processed for storage, wherein in the event that a first data segment included in the portion of the stream currently being processed for storage is determined to be identical to a previously stored data segment, sketches for the plurality of previously stored data segments satisfying the locality criterion are loaded into a limited sketch index;
- and
- a delta compression system having a processor to determine that the first data segment included in the portion of the stream currently being processed for storage is similar to a similar data segment among the data segments in the locale by satisfying a similarity criterion, wherein in the event that the first data segment was determined to be not identical to a previously stored data segment, the similar data segment is determined based at least in part on the limited sketch index, and a reference to the similar data segment and one or more sequences in the first data segment that are not in the similar data segment are stored;
- wherein a master sketch index is accessed only when the sketch of the incoming data segment is not found in the limited sketch index.
1 more element
A method for processing data, comprising:
- selecting a previously stored stream indicated locale, the locale including a selection of a plurality of previously stored data segments satisfying a locality criterion with respect to a portion of a stream currently being processed for storage, wherein in the event that a first data segment included in the portion of the stream currently being processed for storage is determined to be identical to a previously stored data segment, sketches for the plurality of previously stored data segments satisfying the locality criterion are loaded into limited sketch index;
- and
- determining that the first data segment included in the portion of the stream currently being processed for storage is similar to a similar data segment among the data segments in the locale by satisfying a similarity criterion, wherein in the event that the first data segment was determined to be not identical to a previously stored data segment, the similar data segment is determined based at least in part on the limited sketch index, and a reference to the similar data segment and one or more sequences in the first data segment that are not in the similar data segment are stored;
- wherein a master sketch index is accessed only when the sketch of the incoming data segment is not found in the limited sketch index.
1 more element
A computer program product for processing data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
- selecting a previously stored stream indicated locale, the locale including a selection of a plurality of previously stored data segments satisfying a locality criterion with respect to a portion of a stream currently being processed for storage, wherein in the event that a first data segment included in the portion of the stream currently being processed for storage is determined to be identical to a previously stored data segment, sketches for the plurality of previously stored data segments satisfying the locality criterion are loaded into a limited sketch index;
- and
- determining that the first data segment included in the portion of the stream currently being processed for storage is similar to a similar data segment among the data segments in the locale by satisfying a similarity criterion, wherein in the event that the first data segment was determined to be not identical to a previously stored data segment, the similar data segment is determined based at least in part on the limited sketch index, and a reference to the similar data segment and one or more sequences in the first data segment that are not in the similar data segment are stored;
- wherein a master sketch index is accessed only when the sketch of the incoming data segment is not found in the limited sketch index.
1 more element
Dependencies are inferred from the claim text.