Intelligent signature-based anti-cloaking web recrawling

A non-transitory computer-readable medium having instructions stored thereon that are executable by a computing device to perform operations comprising:

• determining, from content returned responsive to multiple requests to a first uniform resource locator (URL) with different requestor profiles, whether the first URL satisfies a first criterion of common cloaking behavior;
• based on a determination that the first URL satisfies the first criterion, generating a first signature for cloaking program code in the content, wherein generating the first signature comprises generating first tokens corresponding to one or more strings in the cloaking program code, wherein the first tokens are standardized across a plurality of programming languages;
• determining whether first tokens in the first signature at least partially match second tokens in one of a plurality of signatures of malicious cloaking program code;
• and
• based on a determination that the first tokens in the first signature at least partially match the second tokens in one of the plurality of signatures, flagging the first URL for recrawling.

An apparatus comprising:

• a processor;
• and
• a computer-readable medium having instructions stored thereon that are executable by the processor to cause the apparatus to,
• send multiple requests with different requestor profiles to each of a set of uniform resource locators (URLs);
• for each URL, compare content received responsive to the requests with different profiles indicates cloaking behavior for the URL;
• for each URL for which cloaking behavior is indicated,
• generate a first signature for cloaking program code in the corresponding content wherein generating the first signature comprises generating first tokens corresponding to one or more strings in the cloaking program code, wherein the first tokens are standardized across a plurality of programming languages;
• determine whether the first tokens in the first signature at least partially match second tokens in one of a plurality of signatures of malicious cloaking program code;
• based on a determination that the first tokens in first signature at least partially match the second tokens in one of the plurality of signatures, indicate the URL for recrawling.