Statistical and neural network approach for data characterization to reduce storage space requirements

A method comprising:

• in a storage system comprising at least one compute node that manages access to non-volatile storage, the compute node configured to respond to commands from host nodes to access host application data stored on the non-volatile storage:
• using a data model that has been trained to recognize compression state to identify the compression state of the host application data stored on the non-volatile storage;
• and
• prompting compression of at least some of the host application data that is identified by the data model as being in an uncompressed state.

A data storage system configured to independently identify compression state of stored data and compress that data, comprising:

• at least one compute node that manages access to non-volatile storage, the compute node configured to respond to commands from host nodes to access host application data stored on the non-volatile storage;
• a data model that has been trained to recognize compression state and encryption state;
• and
• a recommendation engine that:
• uses the data model to identify the compression state and encryption state of data structures or blocks of the host application data stored on the non-volatile storage;
• groups data structures or blocks of the host application data that are uncompressed and unencrypted into at least one group based on similarity of statistical features;
• and
• compresses all data structures or blocks in the group with a single compression algorithm.