Similarity based compression

A non-transitory computer readable medium for compressing an evaluated chunk, the non-transitory computer readable medium stores instructions for:

• calculating similarity hashes for a plurality of chunks;
• wherein one or more of the plurality of chunks are tied to a corresponding reference chunk and one or more of the plurality of chunks are not tied to any corresponding reference chunk;
• calculating a similarity hash for the evaluated chunk;
• determining a most similar chunk of the plurality of chunks based on the similarity hashes of the plurality of chunks and on the calculated similarity hash of the evaluated chunk;
• calculating a first similarity score indicative of a similarity between the most similar chunk and a first reference chunk, wherein the most similar chunk is previously tied to the first reference chunk;
• calculating a second similarity score indicative of a similarity between the most similar chunk and the evaluated chunk, wherein the second similarity score is based on the calculated similarity hash of the evaluated chunk and on a similarity hash of the most similar chunk;
• determining a first relationship between the first similarity score and the second similarity score;
• selecting a compressing scheme (SC) of a set of candidate SCs based on evaluating at least a first candidate SC and a second candidate SC in the set of candidate SCs using at least the first relationship;
• wherein the selected SC is one of:
• the first candidate SC including a compression process that comprises:
• (i) untying the most similar chunk from the first reference chunk
• (ii) producing a delta of the content of the evaluated chunk that is not shared with the most similar chunk by compressing the evaluated chunk based on the most similar chunk, and
• (iii) storing the compressed evaluated chunk that includes the delta and a reference to the most similar chunk;
• and
• the second candidate SC including a compression process that comprises self-compressing the evaluated chunk without tying the evaluated chunk to the most similar chunk;
• and
• compressing the evaluated chunk using the selected SC.

A compute core of a processing system, the compute core is configured to:

• calculate similarity hashes for a plurality of chunks;
• wherein one or more of the plurality of chunks are tied to a corresponding reference chunk and one or more of the plurality of chunks are not tied to any corresponding reference chunk;
• calculate a similarity hash for the evaluated chunk;
• determine a most similar chunk of the plurality of chunks based on the similarity hashes of the plurality of chunks and on the calculated similarity hash of the evaluated chunk;
• calculate a first similarity score indicative of a similarity between the most similar chunk and a first reference chunk, wherein the most similar chunk is previously tied to the first reference chunk;
• calculating a second similarity score indicative of a similarity between the most similar chunk and the evaluated chunk, wherein the second similarity score is based on the calculated similarity hash of the evaluated chunk and on a similarity hash of the most similar chunk;
• determine a first relationship between the first similarity score and the second similarity score;
• select a compressing scheme (SC) of a set of candidate SCs based on evaluating at least a first candidate SC and a second candidate SC in the set of candidate SCs using at least the first relationship;
• wherein the selected SC is one of:
• the first candidate SC including a compression process that comprises:
• (i) untying the most similar chunk from the first reference chunk
• (ii) producing a delta of the content of the evaluated chunk that is not shared with the most similar chunk by compressing the evaluated chunk based on the most similar chunk, and
• (iii) storing the compressed evaluated chunk that includes the delta and a reference to the most similar chunk;
• and
• the second candidate SC including a compression process that comprises self-compressing the evaluated chunk without tying the evaluated chunk to the most similar chunk;
• and
• compress the evaluated chunk using the selected SC.