Method and system for detecting boundaries of data blocks for deduplication

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of deduplicating data objects, the operations comprising:

• receiving a first data stream to be stored in a storage system, the first data stream including a plurality of data blocks, wherein each data block includes a header and a footer;
• determining by a boundary detector boundaries of the data blocks by matching at least a portion of a header with a footer of each data block and matching the at least a portion of the header with a header of an adjacent data block;
• anchoring by an anchoring unit the first data stream based on the determined boundaries of the data blocks using a plurality of anchors;
• deduplicating by a deduplication engine the first data stream into a plurality of deduplicated data segments based on the plurality of anchors;
• and
• storing the deduplicated data segments in one or more persistent storage devices of the storage system.

A storage system, comprising:

• a processor;
• a memory coupled to the processor;
• a boundary detector to receive a first data stream, the first data stream including a plurality of data blocks, wherein each data block includes a header and a footer, and to determine boundaries of the data blocks by matching at least a portion of a header with a footer of each data block and matching the at least a portion of the header with a header of an adjacent data block;
• an anchoring unit to anchor the first data stream based on the determined boundaries of the data blocks using a plurality of anchors;
• a deduplication engine to deduplicate the first data stream into a plurality of deduplicated data segments based on the plurality of anchors;
• and
• one or more storage devices to store the deduplicated data segments.