Selection of digest hash function for different data sets

A system comprising:

• at least one processor;
• and
• a memory comprising code stored thereon that, when executed, perform a method of processing data comprising:
• receiving a plurality of data chunks for a data set;
• performing data deduplication processing for the plurality of data chunks, wherein said data deduplication processing includes:
• determining, using a current hash algorithm, a plurality of digests for the plurality of data chunks of the data set;
• and
• updating a frequency histogram for the data set in accordance the plurality of digests, wherein the frequency histogram is a digest byte value histogram that tracks frequencies or counter values of different observed byte values for each individual byte position in digests computed for chunks of the data set, wherein each of the plurality of digests is a current digest size that is a specified number of bytes, wherein said updating includes performing update processing that updates the frequency histogram for each of the plurality of digests and, for each byte located at each byte position of said each digest, performs processing comprising:
• determining a byte value for said each byte located at said each byte position of said each digest;
• and
• incrementing a cell of the frequency histogram denoting a frequency or counter value corresponding to said byte value for said each byte position;
• determining, in accordance with one or more criteria, whether a frequency distribution of the frequencies or counter values corresponding to the observed byte values for the individual byte positions of the frequency histogram is sufficiently uniform;
• and
• responsive to determining that the frequency distribution of the frequencies or counter values corresponding to the observed byte values for the individual byte positions of the frequency histogram is not sufficiently uniform, performing processing to update data deduplication settings for the data set.

A non-transitory computer readable medium comprising code stored thereon that, when executed, performs a method of processing data comprising:

• receiving a plurality of data chunks for a data set;
• performing data deduplication processing for the plurality of data chunks, wherein said data deduplication processing includes:
• determining, using a current hash algorithm, a plurality of digests for the plurality of data chunks of the data set;
• and
• updating a frequency histogram for the data set in accordance the plurality of digests, wherein the frequency histogram is a digest byte value histogram that tracks frequencies or counter values of different observed byte values for each individual byte position in digests computed for chunks of the data set, wherein each of the plurality of digests is a current digest size that is a specified number of bytes, wherein said updating includes performing update processing that updates the frequency histogram for each of the plurality of digests and, for each byte located at each byte position of said each digest, performs processing comprising:
• determining a byte value for said each byte located at said each byte position of said each digest;
• and
• incrementing a cell of the frequency histogram denoting a frequency or counter value corresponding to said byte value for said each byte position;
• determining, in accordance with one or more criteria, whether a frequency distribution of the frequencies or counter values corresponding to the observed byte values for the individual byte positions of the frequency histogram is sufficiently uniform;
• and
• responsive to determining that the frequency distribution of the frequencies or counter values corresponding to the observed byte values for the individual byte positions of the frequency histogram is not sufficiently uniform, performing processing to update data deduplication settings for the data set.