Techniques for optimizing entropy computations

A method of data processing comprising:

• receiving a data chunk including a plurality of data items;
• performing first processing that computes an entropy value for the data chunk, wherein the first processing includes:
• determining a plurality of counters for data items occurring in the data chunk, wherein a first portion of data items allowable in the data chunk are mapped to the plurality of counters, wherein each of the plurality of counters is associated with a particular data item of the first portion of data items allowable and denotes a current frequency of the particular data item in the data chunk;
• and
• performing second processing using the plurality of counters to determine an entropy value for the data chunk, wherein said second processing includes selecting a precomputed binary logarithmic value from a table for each of the plurality of counters;
• and
• determining, in accordance with the entropy value for the data chunk, whether the data chunk is compressible;
• and
• responsive to determining the data chunk is compressible based on the entropy value for the chunk, compressing the data chunk, wherein the data chunk has a size that is a number of bytes, N, a range is defined from a minimum value of zero and a maximum value of N, and wherein the table includes precomputed binary logarithmic values for each value in the range.

One or more non-transitory computer readable media comprising code stored thereon that, when executed, performs method of data processing comprising:

• receiving a data chunk partitioned into N data items;
• performing first processing that computes an entropy value for the data chunk, wherein the first processing includes:
• determining a plurality of counters for data items occurring in the data chunk, wherein a first portion of data items allowable in the data chunk are mapped to the plurality of counters, wherein each of the plurality of counters is associated with a particular data item of the first portion of data items allowable and denotes a current frequency of the particular data item in the data chunk;
• and
• performing second processing using the plurality of counters to determine an entropy value for the data chunk, wherein said second processing includes selecting a precomputed binary logarithmic value from a table for each of the plurality of counters;
• and
• determining, in accordance with the entropy value for the data chunk, whether the data chunk is compressible;
• and
• responsive to determining the data chunk is compressible based on the entropy value for the chunk, compressing the data chunk, wherein a range is defined from a minimum value of zero and a maximum value of N, wherein N denotes a number of data items in the data chunk, and wherein the table includes precomputed binary logarithmic values for each value in the range.

A system comprising:

• one or more processors;
• and
• a memory comprising code stored thereon that, when executed, performs a method of data processing comprising:
• receiving a data chunk partitioned into N data items;
• performing first processing that computes an entropy value for the data chunk, wherein the first processing includes:
• determining a plurality of counters for data items occurring in the data chunk, wherein a first portion of data items allowable in the data chunk are mapped to the plurality of counters, wherein each of the plurality of counters is associated with a particular data item of the first portion of data items allowable and denotes a current frequency of the particular data item in the data chunk;
• and
• performing second processing using the plurality of counters to determine an entropy value for the data chunk, wherein said second processing includes selecting a precomputed binary logarithmic value from a table for each of the plurality of counters;
• and
• determining, in accordance with the entropy value for the data chunk, whether the data chunk is compressible;
• and
• responsive to determining the data chunk is compressible based on the entropy value for the chunk, compressing the data chunk, wherein a range is defined from a minimum value of zero and a maximum value of N, wherein N denotes a number of data items in the data chunk, and wherein the table includes precomputed binary logarithmic values for each value in the range.

One or more non-transitory computer readable media comprising code stored thereon that, when executed, performs a method of data processing comprising:

• receiving a data chunk including a plurality of data items;
• performing first processing that computes an entropy value for the data chunk, wherein the first processing includes:
• determining a plurality of counters for data items occurring in the data chunk, wherein a first portion of data items allowable in the data chunk are mapped to the plurality of counters, wherein each of the plurality of counters is associated with a particular data item of the first portion of data items allowable and denotes a current frequency of the particular data item in the data chunk;
• and
• performing second processing using the plurality of counters to determine an entropy value for the data chunk, wherein said second processing includes selecting a precomputed binary logarithmic value from a table for each of the plurality of counters;
• and
• determining, in accordance with the entropy value for the data chunk, whether the data chunk is compressible;
• and
• responsive to determining the data chunk is compressible based on the entropy value for the chunk, compressing the data chunk, wherein the data chunk has a size that is a number of bytes, N, a range is defined from a minimum value of zero and a maximum value of N, and wherein the table includes precomputed binary logarithmic values for each value in the range.

A system comprising:

• one or more processors;
• and
• a memory comprising code stored thereon that, when executed, performs a method of data processing comprising:
• receiving a data chunk including a plurality of data items;
• performing first processing that computes an entropy value for the data chunk, wherein the first processing includes:
• determining a plurality of counters for data items occurring in the data chunk, wherein a first portion of data items allowable in the data chunk are mapped to the plurality of counters, wherein each of the plurality of counters is associated with a particular data item of the first portion of data items allowable and denotes a current frequency of the particular data item in the data chunk;
• and
• performing second processing using the plurality of counters to determine an entropy value for the data chunk, wherein said second processing includes selecting a precomputed binary logarithmic value from a table for each of the plurality of counters;
• and
• determining, in accordance with the entropy value for the data chunk, whether the data chunk is compressible;
• and
• responsive to determining the data chunk is compressible based on the entropy value for the chunk, compressing the data chunk, wherein the data chunk has a size that is a number of bytes, N, a range is defined from a minimum value of zero and a maximum value of N, and wherein the table includes precomputed binary logarithmic values for each value in the range.