Efficient segment detection for deduplication
AssigneeDell Products L.P.
InventorVinod Jayaraman
Life of this patent
Estimating the term…
Abstract
Mechanisms are provided for efficiently detecting segments for deduplication. Data is analyzed to determine file types and file components. File types such as images may have optimal data segment boundaries set at the file boundaries. Other file types such as container files are delayered to extract objects to set optimal data segment boundaries based on file type or based on the boundaries of the individual objects. Storage of unnecessary information is minimized in a deduplication dictionary while allowing for effective deduplication.
Claims20
A method comprising:
- determining that a first file received for deduplication at a deduplication system is a non-container file based on file type information associated with the first file;
- identifying whether the first file would benefit from fine-grained segmentation based on file type information;
- determining that a second file received for deduplication at the deduplication system is a container file based on file type information associated with the second file;
- delayering the second file to extract a plurality of components;
- setting segment boundaries for deduplication of the second file at component boundaries, wherein segment boundaries are used during deduplication to identify duplicate segments.
A deduplication system comprising:
- an interface configured to receive a first file and a second file for deduplication at the deduplication system;
- a processor configured to determine that the first file is a non-container file and the second file is a container file, wherein the processor determines that the first file would benefit from fine-grained segmentations based on file type information associated with the first file;
- wherein the processor delayers the second file to extract a plurality of components and sets segment boundaries for deduplication of the second file at component boundaries, wherein segment boundaries are used during deduplication to identify duplicate segments.
A non-transitory computer readable medium comprising:
- computer code for determining that a first file received for deduplication at a deduplication system is a non-container file based on file type information associated with the first file;
- computer code for identifying whether the first file would benefit from fine-grained segmentation based on file type information;
- computer code for determining that a second file received for deduplication at the deduplication system is a container file based on file type information associated with the second file;
- computer code for delayering the second file to extract a plurality of components;
- computer code for setting segment boundaries for deduplication of the second file at component boundaries, wherein segment boundaries are used during deduplication to identify duplicate segments.
Dependencies are inferred from the claim text.