Systems and methods for efficient data searching, storage and reduction

A non-transitory computer-readable medium encoded with computer executable instructions that cause a computer to perform a method of searching in repository data for data that are similar to an input data, wherein the repository data is divided into one or more repository data chunks and the input data comprises a plurality of input data chunks, the medium comprising:

• program code for generating a corresponding set of repository distinguishing characteristics (RDCs) for each repository data chunk;
• program code for generating a set of input distinguishing characteristics (IDCs) for each input data chunk;
• and program code for searching for data in the repository data that is similar to the input data by comparing the IDCs and RDCs,
• wherein the program code for generating each set of IDCs and the program code for generating each set of RDCs comprises:
• program code for applying a hash function to the respective input data chunk or repository data chunk to generate a plurality of hashes, each hash comprising a hash value and a hash position within the data chunk;
• program code for applying a first function to the plurality of generated hashes to identify a first subset of hashes distributed across the data chunk;
• program code for applying a second function to the hash positions of the hashes of the first subset to identify a second subset of the plurality of generated hashes;
• and program code for defining the second subset of hashes as the set of respective IDCs or RDCs.

A system for searching in repository data for data that are similar to an input data, wherein the repository data is divided into one or more repository data chunks and the input data comprises a plurality of input data chunks, the system comprising:

• a mechanism for, for each repository data chunk, generating a corresponding set of repository distinguishing characteristics (RDCs);
• a mechanism for generating a corresponding set of input distinguishing characteristics (IDCs) for each input data chunk;
• and
• a mechanism for searching for data in the repository data that is similar to the input data by comparing the IDCs and RDCs,
• wherein the mechanism for generating the IDCs and the mechanism for generating the RDCs each comprises:
• a mechanism for applying a hash function to the respective input data chunk or repository data chunk to generate a plurality of hashes, each hash comprising a hash value and a hash position within the data chunk;
• a mechanism for applying a first function to the plurality of generated hashes to identify a first subset of hashes distributed across the data chunk;
• a mechanism for applying a second function to the hash positions of the hashes of the first subset to identify a second subset of the plurality of generated hashes;
• and
• a mechanism for defining the second subset of hashes as the set of respective IDCs or RDCs.