Systems and methods for efficient data searching, storage and reduction

A computer-implemented method, comprising:

• receiving, by a computer, input data that includes input data chunks;
• for each input data chunk, generating, by the computer, a corresponding set of input distinguishing characteristics (IDCs);
• and
• wherein each set of IDCs is generated by:
• applying, by the computer, a hash function to the respective input data chunk or repository data chunk to generate a plurality of hashes, each hash comprising a hash value and a hash position within the data chunk;
• applying, by the computer, a first function to the plurality of generated hashes to identify a first subset of hashes distributed across the data chunk;
• applying, by the computer, a second function to the hash positions of the hashes of the first subset to identify a second subset of the plurality of generated hashes;
• and
• defining, by the computer, the second subset of hashes as the set of IDCs;
• searching, by the computer, for data in repository data that is similar to input data by comparing the IDCs to repository distinguishing characteristics (RDCs) corresponding to the repository data, wherein comparing the generated IDCs and the RDCs reduces an amount of memory consumed during the searching;
• and
• outputting, by the computer, a result of the searching.

A computer program product, the computer program product comprising a computer readable storage medium having program instructions executable by a computer to cause the computer to perform a method comprising:

• for each repository data chunk in repository data that comprises a plurality of the repository data chunks, generating, by the computer, a corresponding set of repository distinguishing characteristics (RDCs),
• wherein each set of RDCs is generated by:
• applying, by the computer, a hash function to a respective input data chunk or repository data chunk to generate a plurality of hashes, each hash comprising a hash value and a hash position within the data chunk;
• applying, by the computer, a first function to the plurality of generated hashes to identify a first subset of hashes distributed across the data chunk;
• applying, by the computer, a second function to the hash positions of the hashes of the first subset to identify a second subset of the plurality of generated hashes;
• and
• defining, by the computer, the second subset of hashes as the set of RDCs;
• searching, by the computer, for data in the repository data that is similar to input data by comparing the RDCs to input distinguishing characteristics (IDCs) corresponding to the input data, wherein comparing the generated RDCs and the IDCs reduces an amount of memory consumed during the searching;
• and
• outputting, by the computer, a result of the searching.