# US 11,444,977 B2 · Intelligent signature-based anti-cloaking web recrawling

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,444,977 B2
- Document ID: US11444977B2
- Status: Granted
- Kind: B2
- Application number: 16/659,917
- Filed: 2019-10-22
- Published: 2021-04-22
- Granted: 2022-09-13
- Earliest claimed: 2019-10-22
- Assignees: Palo Alto Networks, Inc.
- Applicants: Palo Alto Networks, Inc.
- Inventors: Oleksii Starov; Zhanhao Chen; Yuchen Zhou; Fang Liu
- Web page: https://patentgrep.com/patent/US11444977B2

## CPC

- H04L 63/14: {for detecting or protecting against malicious traffic}
- H04L 63/1483: {service impersonation, e.g. phishing, pharming or web spoofing (detection of rogue wireless access points H04W12/12)}
- G06F 16/2379: {Updates performed during online database operations; commit processing}
- G06F 16/951: Indexing; Web crawling techniques
- G06F 16/972: {Access to data in other repository systems, e.g. legacy data or dynamic Web page generation}
- G06F 21/563: {by source code analysis}
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- G06F 2221/033: Test or assess software
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Web sites are crawled using multiple browser profiles to avoid malicious cloaking. Based on web page content returned from HTTP requests using the multiple browser profiles, web sites returning substantively different content to HTTP requests for different browser profiles are identified. Web sites are further filtered by common cloaking behavior, and redirect scripts are extracted from web page content that performed cloaking. Signatures comprising tokenized versions of the redirect scripts are generated and compared to a database of known cloaking signatures. URLs corresponding to signatures having approximate matches with signatures in the database are flagged for recrawling. Recrawled URLs are verified for malicious cloaking again using HTTP requests from multiple browser profiles.

## Claims

### Claim 1

A method comprising:
identifying a first uniform resource locator (URL) at which cloaking program code is being used based, at least in part, on different content returned responsive to multiple requests to the first URL with different requestor profiles;
generating a first signature of the cloaking program code in the content, wherein generating the first signature comprises generating first tokens corresponding to one or more strings in the cloaking program code, wherein the first tokens are standardized across a plurality of programming languages;
determining whether the first tokens in the first signature at least partially match second tokens in one of a plurality of signatures of malicious cloaking program code; and
based on a determination that the first tokens in the first signature at least partially match the second tokens in one of the plurality of signatures, flagging the first URL for recrawling.

### Claim 2 (depends on 1)

The method of claim 1 further comprising recrawling with at least two different requestor profiles a set of URLs flagged for recrawling including the first URL and verifying that cloaking program code is being used at the first URL based on the recrawling.

### Claim 3 (depends on 2)

The method of claim 2 further comprising updating a database of malicious cloaking program code signatures with the first signature based on verification that cloaking program code is being used at the first URL.

### Claim 4 (depends on 1)

The method of claim 1 further comprising:
selecting a first subset of URLs from a plurality of URLs to be crawled, wherein the first subset of URLs includes the first URL; and
for each of the first subset of URLs, requesting content with at least two different requestor profiles.

### Claim 5 (depends on 4)

The method of claim 4, wherein selecting the first subset of URLs comprises arbitrarily selecting the first subset of URLs.

### Claim 6 (depends on 1)

The method of claim 1, wherein flagging the first URL for recrawling is based, at least in part, on a fuzzy matching distance between the first tokens in the first signature and the second tokens in the one of the plurality of signatures of malicious cloaking program code.

### Claim 7 (depends on 1)

The method of claim 1, further comprising, for each request to the first URL with a different requestor profile, storing in a website repository a digital footprint of content returned responsive to the request to the first URL.

### Claim 8

A non-transitory computer-readable medium having instructions stored thereon that are executable by a computing device to perform operations comprising:
determining, from content returned responsive to multiple requests to a first uniform resource locator (URL) with different requestor profiles, whether the first URL satisfies a first criterion of common cloaking behavior;
based on a determination that the first URL satisfies the first criterion, generating a first signature for cloaking program code in the content, wherein generating the first signature comprises generating first tokens corresponding to one or more strings in the cloaking program code, wherein the first tokens are standardized across a plurality of programming languages;
determining whether first tokens in the first signature at least partially match second tokens in one of a plurality of signatures of malicious cloaking program code; and
based on a determination that the first tokens in the first signature at least partially match the second tokens in one of the plurality of signatures, flagging the first URL for recrawling.

### Claim 9 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the instructions executable by the computing device further comprise instructions to recrawl with at least two different requestor profiles a set of URLs flagged for recrawling including the first URL and verify that cloaking program code is being used at the first URL based on the recrawling.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instructions executable by the computing device further comprise instructions to update a database of malicious cloaking program code signatures with the first signature based on verification that cloaking program code is being used at the first URL.

### Claim 11 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the instructions executable by the computing device further comprise instructions to:
select a first subset of URLs from a plurality of URLs to be crawled, wherein the first subset of URLs includes the first URL; and
for each of the first subset of URLs, request content with at least two different requestor profiles.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein the instructions executable by the computing device to select the first subset of URLs comprise instructions to arbitrarily select the first subset of URLs.

### Claim 13 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the instructions executable by the computing device to flag the first URL for recrawling comprise instructions executable by the computing device to flag the first URL for recrawling based, at least in part, on a fuzzy matching distance between the first tokens in the first signature and the second tokens in the one of the plurality of signatures of malicious cloaking program code.

### Claim 14 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the instructions executable by the computing device further comprise instructions to, for each request to the first URL with a different requestor profile, store in a website repository a digital footprint of content returned responsive to the request to the first URL.

### Claim 15

An apparatus comprising:
a processor; and
a computer-readable medium having instructions stored thereon that are executable by the processor to cause the apparatus to,
send multiple requests with different requestor profiles to each of a set of uniform resource locators (URLs);
for each URL, compare content received responsive to the requests with different profiles indicates cloaking behavior for the URL;
for each URL for which cloaking behavior is indicated,
generate a first signature for cloaking program code in the corresponding content wherein generating the first signature comprises generating first tokens corresponding to one or more strings in the cloaking program code, wherein the first tokens are standardized across a plurality of programming languages;
determine whether the first tokens in the first signature at least partially match second tokens in one of a plurality of signatures of malicious cloaking program code;
based on a determination that the first tokens in first signature at least partially match the second tokens in one of the plurality of signatures, indicate the URL for recrawling.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the instructions stored on the computer-readable medium further comprise instructions executable by the processor to cause the apparatus to update a database of malicious cloaking program code signatures with the first signature based on verification that cloaking program code is being used at the URL.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein the instructions executable by the processor to cause the apparatus to flag the URL for recrawling comprise instructions to flag the URL for recrawling based, at least in part, on a fuzzy matching distance between the first tokens in the first signature and the second tokens in the one of the plurality of signatures of malicious cloaking program code.

### Claim 18 (depends on 1)

The method of claim 1, wherein the plurality of signatures of the malicious cloaking program code were generated from second tokens corresponding to one or more strings in the malicious cloaking program code, wherein the second tokens are standardized across the plurality of programming languages, wherein the first tokens and second tokens indicate, at least in part, functionality of the one or more strings in the cloaking program code and the one or more strings in the malicious cloaking program code, respectively.

### Claim 19 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the plurality of signatures of the malicious program code were generated from second tokens corresponding to one or more strings in the malicious cloaking program code, wherein the second tokens are standardized across the plurality of programming languages, wherein the first tokens and second tokens indicate, at least in part, functionality of the one or more strings in the cloaking program code and the one or more strings in the malicious cloaking program code, respectively.

### Claim 20 (depends on 15)

The apparatus of claim 15, wherein the plurality of signatures of the malicious cloaking program code were generated from second tokens corresponding to one or more strings in the malicious cloaking program code, wherein the second tokens are standardized across the plurality of programming languages, wherein the first tokens and second tokens indicate, at least in part, functionality of the one or more strings in the cloaking program code and the one or more strings in the malicious cloaking program code, respectively.