# US 2013/0159262 A1 · Efficient segment detection for deduplication > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2013/0159262 A1 - Document ID: US20130159262A1 - Status: Published application - Kind: A1 - Application number: 13/760,131 - Filed: 2013-02-06 - Published: 2013-06-20 - Earliest claimed: 2009-11-23 - Assignees: Dell Products L.P. - Inventors: Vinod Jayaraman - Web page: https://patentgrep.com/patent/US20130159262A1 ## CPC - G06F 16/1748: {De-duplication implemented within the file system, e.g. based on file segments (de-duplication techniques in storage systems for the management of data blocks G06F3/0641)} - G06F 16/1752: {based on file chunks} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Mechanisms are provided for efficiently detecting segments for deduplication. Data is analyzed to determine file types and file components. File types such as images may have optimal data segment boundaries set at the file boundaries. Other file types such as container files are delayered to extract objects to set optimal data segment boundaries based on file type or based on the boundaries of the individual objects. Storage of unnecessary information is minimized in a deduplication dictionary while allowing for effective deduplication. ## Claims ### Claim 1 A method comprising: determining that a first file received for deduplication at a deduplication system is a non-container file based on file type information associated with the first file; identifying whether the first file would benefit from fine-grained segmentation based on file type information; determining that a second file received for deduplication at the deduplication system is a container file based on file type information associated with the second file; delayering the second file to extract a plurality of components; setting segment boundaries for deduplication of the second file at component boundaries, wherein segment boundaries are used during deduplication to identify duplicate segments. ### Claim 2 (depends on 1) The method of claim 1, wherein the second file is recursively parsed to determine whether the plurality of components include container files. ### Claim 3 (depends on 1) The method of claim 1, wherein segment sizes are based on a dynamically tunable value if the first file would benefit from fine-grained segmentation. ### Claim 4 (depends on 1) The method of claim 1, wherein segment sizes are based on a predetermined value based on file type if the first file would benefit from fine-grained segmentation. ### Claim 5 (depends on 1) The method of claim 1, wherein the second file includes a plurality of container and non-container components. ### Claim 6 (depends on 1) The method of claim 1, wherein the second file includes a plurality of non-container components that would benefit from fine-grained segmentation. ### Claim 7 (depends on 1) The method of claim 1, wherein container files include a plurality of components. ### Claim 8 (depends on 7) The method of claim 7, wherein a plurality of files including the first file and the second file are deduplicated by generating a plurality of filemaps. ### Claim 9 (depends on 7) The method of claim 7, wherein a plurality of files including the first file and the second file are deduplicated by generating a plurality of datastore suitcases. ### Claim 10 (depends on 9) The method of claim 9, wherein the plurality of datastore suitcases comprise a plurality of reference counts corresponding to a plurality of deduplicated data segments. ### Claim 11 A deduplication system comprising: an interface configured to receive a first file and a second file for deduplication at the deduplication system; a processor configured to determine that the first file is a non-container file and the second file is a container file, wherein the processor determines that the first file would benefit from fine-grained segmentations based on file type information associated with the first file; wherein the processor delayers the second file to extract a plurality of components and sets segment boundaries for deduplication of the second file at component boundaries, wherein segment boundaries are used during deduplication to identify duplicate segments. ### Claim 12 (depends on 11) The deduplication system of claim 11, wherein the second file is recursively parsed to determine whether the plurality of components include container files. ### Claim 13 (depends on 11) The deduplication system of claim 11, wherein segment sizes are based on a dynamically tunable value if the first file would benefit from fine-grained segmentation. ### Claim 14 (depends on 11) The deduplication system of claim 11, wherein segment sizes are based on a predetermined value based on file type if the first file would benefit from fine-grained segmentation. ### Claim 15 (depends on 11) The deduplication system of claim 11, wherein the second file includes a plurality of container and non-container components. ### Claim 16 (depends on 11) The deduplication system of claim 11, wherein the second file includes a plurality of non-container components that would benefit from fine-grained segmentation. ### Claim 17 (depends on 11) The deduplication system of claim 11, wherein container files include a plurality of components. ### Claim 18 (depends on 17) The deduplication system of claim 17, wherein a plurality of files including the first file and the second file are deduplicated by generating a plurality of filemaps. ### Claim 19 (depends on 17) The deduplication system of claim 17, wherein a plurality of files including the first file and the second file are deduplicated by generating a plurality of datastore suitcases. ### Claim 20 A non-transitory computer readable medium comprising: computer code for determining that a first file received for deduplication at a deduplication system is a non-container file based on file type information associated with the first file; computer code for identifying whether the first file would benefit from fine-grained segmentation based on file type information; computer code for determining that a second file received for deduplication at the deduplication system is a container file based on file type information associated with the second file; computer code for delayering the second file to extract a plurality of components; computer code for setting segment boundaries for deduplication of the second file at component boundaries, wherein segment boundaries are used during deduplication to identify duplicate segments.