System and method for global data compression

A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process, the process comprising:

• splitting a dataset into a plurality of blocks;
• for each block of the plurality of blocks:
• computing at least one similarity hash for the block;
• determining, based on the at least one similarity hash, whether a similar block is found for the block, wherein a similar block for a block has a similarity hash that is similar to one of the computed at least one similarity hash for the block;
• compressing the block by replacing data of the block with a reference to the similar block and a delta when a similar block is found, wherein the delta is a difference in data between the block and the similar block;
• and
• compressing the block independently when a similar block is not found.

A system for global data compression, comprising:

• a processing circuitry;
• and
• a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:
• split a dataset into a plurality of blocks;
• for each block of the plurality of blocks:
• compute at least one similarity hash for the block;
• determine, based on the at least one similarity hash, whether a similar block is found for the block, wherein a similar block for a block has a similarity hash that is similar to one of the computed at least one similarity hash for the block;
• compress the block by replacing data of the block with a reference to the similar block and a delta when a similar block is found, wherein the delta is a difference in data between the block and the similar block;
• and
• compress the block independently when a similar block is not found.