Consistently grouping and routing data segments for deduplication

A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:

• create a first sequence of data segments by consecutively grouping sequential data segments from a stream of data until a size of the grouped sequential data segments is larger than a minimum size;
• create a second sequence of data segments by consecutively grouping sequential data segments that immediately follow the first sequence of data segments in the data stream until an aggregate of a size of the grouped sequential data segments that immediately follow the first sequence of data segments and the size of the first sequence of data segments is larger than a maximum size, the second sequence of data segments being distinct from the first sequence of data segments;
• determine a feature value for each data segment in the second sequence of data segments;
• select a value from the feature values, and a data segment corresponding to the selected value from the second sequence of data segments;
• determine an additional feature value for at least one data segment in the next sequence of data segments;
• divide the next sequence of data segments at the selected data segment into an ending part of the initial group sequence of data segments and a beginning part of the next group sequence of data segments;
• combine the beginning part of the initial group sequence of data segments with the ending part of the initial sequence of data segments from the next sequence of data segments in the stream of data being routed for deduplication, the combination forming an initial group of data segments;
• determine a similarity group identifier based on one of the at least one additional feature values,
• route, based on the similarity group identifier, one of the initial group of data segments or a group of corresponding fingerprints for deduplication to one of a plurality of deduplication modules that are each uniquely responsible for corresponding groups of data segments associated with a range of similarity group identifiers including the determined similarity group identifier;
• and
• deduplicate, by the one of the plurality of deduplication modules, one of the group of fingerprints or the corresponding initial group of data segments being similar to, but not identical to, the groups of data segments associated with the one of the plurality of deduplication modules.

A computer-implemented method for consistently grouping and routing data segments for deduplication, comprising:

• creating a first sequence of data segments by consecutively grouping sequential data segments from a stream of data until a size of the grouped sequential data segments larger than a minimum size;
• creating a second sequence of data segments by consecutively grouping sequential data segments that immediately follow the first sequence of data segments in the data stream until an aggregate of a size of the grouped sequential data segments that immediately follow the first sequence of data segments and the size of the first sequence of data segments is larger than a maximum size, the second sequence of data segments being distinct from the first sequence of data segments;
• determining a feature value for each data segment in the second sequence of data segments;
• selecting a value from the feature values, and a data segment corresponding to the selected value from the second sequence of data segments;
• determining an additional feature value for at least one data segment in the next sequence of data segments;
• dividing the next sequence of data segments at the selected data segment into an ending part of the initial group sequence of data segments and a beginning part of the next group sequence of data segments;
• combining the beginning part of the initial group sequence of data segments with the ending part of the initial sequence of data segments from the next sequence of data segments in the stream of data being routed for deduplication, the combination forming an initial group of data segments;
• determining a similarity group identifier based on one of the at least one additional feature values,
• routing, based on the similarity group identifier, one of the initial group of data segments or a group of corresponding fingerprints for deduplication to one of a plurality of deduplication modules that are each uniquely responsible for corresponding groups of data segments associated with a range of similarity group identifiers including the determined similarity group identifier;
• and
• deduplicating, by the one of the plurality of deduplication modules, one of the group of fingerprints or the corresponding initial group of data segments being similar to, but not identical to, the groups of data segments associated with the one of the plurality of deduplication modules.