Fragmented malware hash lookup in cloud repository

An apparatus comprising:

• a network interface unit to communicate over a network with a data inspection device having access to at least portions of a data file of interest;
• and
• a processor of a server device coupled to the network interface unit and configured to assist the data inspection device with matching the data file to known data files represented on the server device, wherein the processor is configured to:
• construct for each known data file a respective hash tree as a multi-level hash tree including leaf nodes that occupy a lowest level and non-leaf nodes that occupy ascending levels leading to a root node, which occupies a highest level, wherein the processor is configured to:
• fragment the known data file into contiguous spaced fragments;
• select from the contiguous spaced fragments separated one from the next by a number of skipped fragments defined by an offset schema;
• and
• generate nodes of the hash tree based on hashes of the contiguous spaced fragments, but not the skipped fragments;
• receive from the data inspection device a hash of a fragment of the data file of interest and an indication of a level of a hash tree to which the hash corresponds;
• compare the hash to nodes at the level in each of the hash trees constructed using the offset schema;
• and
• send compare results to the data inspection device indicating a match or a mismatch between the hash and the hash trees.

A method comprising:

• at a data inspection device configured to match a data file of interest including contiguous fragments of data with known data files represented as respective hash trees on a server device configured to communicate with the data inspection device over a network:
• sequentially processing successive spaced fragments that are selected from the contiguous fragments so as to be spaced one from the next by a number of skipped fragments defined by an offset schema, the sequentially processing including, for each spaced fragment:
• generating a hash based on the spaced fragment, but not any skipped fragments, such that iterations of the generating the hash in constructing at least a portion of a hash tree of the data file of interest as a multi-level hash tree including leaf nodes that occupy a lowest level and non-leaf nodes that occupy ascending levels leading to a root node, which occupies a highest level;
• sending the hash, an indication of a level of the hash tree to which the hash corresponds, and the offset schema to the server device;
• and
• receiving from the server device compare results that indicate a match or a mismatch between the hash and nodes at the level in each of all hash trees on the server device that are based on the offset schema.