System and method for preprocessing a data set to improve deduplication

A system for detecting duplicate data in an input data stream, the system comprising:

• a processor;
• a stream folder, operatively coupled to the processor, configured to receive an input data stream and to output a folded stream, wherein a first segment of the input data stream is excluded from the folded stream, and wherein a second segment of the input data stream is converted by a logical operation to reduce a length of the second segment before inclusion in the folded stream;
• and
• a stream duplicate detector, operatively coupled to the processor, configured to receive the folded stream and the input data stream, wherein the stream duplicate detector detects duplicate data in the input data stream by utilizing the folded stream, wherein the stream duplicate detector detects duplicate data by identifying an anchor in the folded stream and mapping the anchor into the input data stream.

A storage server comprising:

• a network adaptor through which to communicate over a network;
• a storage adaptor through which to access a mass storage facility;
• a processor;
• and
• a memory storing instructions which, upon execution by the processor, cause the storage server to:
• identify a plurality of segments in an input data set received via the network adaptor;
• generate from the input data set an output data set that includes at least one but not all of the plurality of segments in the input data set;
• process the output data set with a logical operator to reduce a length of each segment in the output data set;
• remove duplicate data from the input data set by utilizing the output data set, wherein the duplicate data is removed from the input data set by identifying an anchor in the output data set and mapping the anchor into the input data set;
• and
• store the input data set on the mass storage facility via the storage adaptor.

A method for detecting duplicate data in a data set at a storage server, the method comprising:

• partitioning a data set into a plurality of segments;
• performing a logical operation on at least one, but not all, of the plurality of segments to shorten a length of the at least one of the plurality of segments prior to detecting duplicate data in the data set;
• and
• detecting at the storage server duplicate data in the data set by utilizing the at least one of the plurality of segments, wherein the detecting the duplicate data includes:
• identifying anchors in the at least one of the plurality of segments;
• and
• mapping the anchors into the data set.

A method of detecting duplicate data in an input data set, the method comprising:

• identifying a plurality of segments in an input data set;
• generating from the input data set an output data set that includes at least one but not all of the plurality of segments in the input data set;
• processing the output data set with a logical operator to reduce a length of each segment in the output data set;
• and
• detecting duplicate data in the input data set by utilizing the output data set, wherein detecting the duplicate data includes:
• identifying an anchor in the output data set;
• and
• mapping the anchor into the input data set.

A system for detecting duplicate data in an input data set, the system comprising:

• a processor;
• means for identifying, operatively coupled to the processor, a plurality of segments in an input data set;
• means for generating, operatively coupled to the processor, from the input data set an output data set that includes at least one but not all of the plurality of segments in the input data set;
• means for processing, operatively coupled to the processor, the output data set with a logical operator to reduce a length of each segment in the output data set;
• and
• means for detecting, operatively coupled to the processor, duplicate data in the input data set by utilizing the output data set, wherein the means for detecting the duplicate data include:
• means for identifying, operatively coupled to the processor, an anchor in the output data set;
• and
• means for mapping, operatively coupled to the processor, the anchor into the input data set.