Back to results
US 8,751,448 B1Granted
State-based directing of segments in a multinode deduplicated storage system
AssigneeEMC Corporation
InventorsFrederick Douglis, Philip Shilane, R. Hugo Patterson
Life of this patent
Estimating the term…
Abstract
A system for directing for storage includes a processor and a memory. The processor is configured to determine a segment overlap for each of a plurality of nodes. The processor is further configured to determine a selected node of the plurality of nodes based at least in part on the segment overlap for each of the plurality of nodes and based at least in part on a selection criteria. The memory is coupled to the processor and configured to provide the processor with instructions.
Claims17
A system for directing for storage, comprising:
- a processor configured to:
- segment input data comprising a data stream, a data block, or a data file into a plurality of segments;
- group at least some of the plurality of segments into a super segment including by:
- determining, for each of at least some of the plurality of segments, whether a hash value computed for a segment of the plurality of segments that is associated with a segment boundary and for the segment boundary meet one or more super segment boundary criteria;
- and
- in the event that the one or more super segment boundary criteria are met, setting a super segment boundary at the segment boundary that is associated with the segment;
- and
- determine, for a first node of a plurality of nodes, a segment overlap for the super segment by determining whether each of the at least some of the plurality of segments comprising the super segment is stored on the first node based at least in part on a probabilistic data structure, wherein the segment overlap comprises a value representative of a number of the plurality of segments comprising the super segment that have been previously stored on the first node of the plurality of nodes;
- determine a selected node of the plurality of nodes including by comparing the segment overlap determined for at least the first node to a threshold, wherein the threshold is a value that is based at least in part on a total number of the plurality of segments comprising the super segment and a total number of nodes in the plurality of nodes;
- and
- a memory coupled to the processor and configured to provide the processor with instructions.
8 more elements
A method for directing for storage, comprising:
- segmenting input data comprising a data stream, a data block, or a data file into a plurality of segments;
- grouping at least some of the plurality of segments into at least one super segment including by:
- determining, for each of at least some of the plurality of segments, whether a hash value computed for a segment associated with a segment boundary and the segment boundary meet one or more super segment boundary criterion;
- and
- setting a super segment boundary at a segment boundary associated with a segment determined to meet the super segment boundary criterion;
- determining, using a processor, for a first node of a plurality of nodes, a segment overlap for the at least one super segment including by determining whether each of at least a subset of the plurality of segments comprising the super segment is stored on the first node based at least in part on a probabilistic data structure, wherein the segment overlap comprises a value representative of a number of the plurality of segments comprising the at least one super segment that have been previously stored on the first node of the plurality of nodes;
- and
- determining a selected node of the plurality of nodes including by comparing the segment overlap determined for at least the first node to a threshold, wherein the threshold is a value that is based at least in part on a total number of the plurality of segments comprising the at least one super segment and a total number of nodes in the plurality of nodes.
5 more elements
A computer program product for directing for storage, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
- segmenting input data comprising a data stream, a data block, or a data file into a plurality of segments;
- grouping at least some of the plurality of segments into at least one super segment including by:
- determining, for each of at least some of the plurality of segments, whether a hash value computed for a segment associated with a segment boundary and the segment boundary meet one or more super segment boundary criterion;
- and
- setting a super segment boundary at a segment boundary associated with a segment determined to meet the super segment boundary criterion;
- determining, for a first node of a plurality of nodes, a segment overlap for the at least one super segment including by determining whether each of at least a subset of the plurality of segments comprising the super segment is stored on the first node based at least in part on a probabilistic data structure, wherein the segment overlap comprises a value representative of a number of the plurality of segments comprising the at least one super segment that have been previously stored on the first node of the plurality of nodes;
- and
- determining a selected node of the plurality of nodes including by comparing the segment overlap determined for at least the first node to a threshold, wherein the threshold is a value that is based at least in part on a total number of the plurality of segments comprising the at least one super segment and a total number of nodes in the plurality of nodes.
5 more elements
Dependencies are inferred from the claim text.