Facet Whitelisting in Anomaly Detection

A non-transitory computer-readable storage medium storing computer program instructions executable by a processor to perform operations comprising:

• receiving a first full hash and a first set of subhashes from a client, wherein the first full hash is a hash of an entire first file and each subhash in the first set of subhashes is a hash of a facet of the first file, wherein a file comprises a code portion and a non-code portion and a facet is at least part of the non-code portion of the file;
• determining whether the first full hash is whitelisted;
• responsive to determining the first full hash is whitelisted, updating, for each subhash in the first set of subhashes, an associated clean count, wherein the clean count tracks a historic number of whitelisted files with which the subhash is associated;
• responsive to a first clean count of the clean counts exceeding a threshold clean count, adding the subhash associated with the first clean count to a subhash whitelist;
• receiving a second full hash and a second set of subhashes from the client, wherein the second full hash is a hash of an entire second file and each subhash in the second set of subhashes is a hash of a facet of the second file;
• determining whether the second full hash is whitelisted;
• responsive to determining the second full hash is not whitelisted, determining whether at least one subhash in the second set of subhashes is included in the subhash whitelist;
• responsive to determining at least one subhash in the second set of subhashes is included in the subhash whitelist, determining the second file is whitelisted;
• and
• reporting that the second file is whitelisted to the client.

A system, comprising:

• a processor;
• and
• a non-transitory computer-readable storage medium storing computer program instructions executable by the processor to perform operations comprising:
• receiving a first full hash and a first set of subhashes from a client, wherein the first full hash is a hash of an entire first file and each subhash in the first set of subhashes is a hash of a facet of the first file, wherein a file comprises a code portion and a non-code portion and a facet is at least part of the non-code portion of the file;
• determining whether the first full hash is whitelisted;
• responsive to determining the first full hash is whitelisted, updating, for each subhash in the first set of subhashes, an associated clean count, wherein the clean count tracks a historic number of whitelisted files with which the subhash is associated;
• responsive to a first clean count of the clean counts exceeding a threshold clean count, adding the subhash associated with the first clean count to a subhash whitelist;
• receiving a second full hash and a second set of subhashes from the client, wherein the second full hash is a hash of a second file and each subhash in the second set of subhashes is a hash of a facet of the second file;
• determining whether the second full hash is whitelisted;
• responsive to determining the second full hash is not whitelisted, determining whether at least one subhash in the second set of subhashes is included in the subhash whitelist;
• responsive to determining at least one subhash in the second set of subhashes is included in the subhash whitelist, determining the second file is whitelisted;
• and
• reporting that the second file is whitelisted to the client.