# US 2020/0007559 A1 · Web Threat Investigation Using Advanced Web Crawling

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0007559 A1
- Document ID: US20200007559A1
- Status: Published application
- Kind: A1
- Application number: 16/021,630
- Filed: 2018-06-28
- Published: 2020-01-02
- Earliest claimed: 2018-06-28
- Assignees: Webroot Inc.
- Applicants: Webroot Inc.
- Inventors: John Morgan; Rob Tietje; David Wang; Tejas Pattabhi; Josh Biol
- Web page: https://patentgrep.com/patent/US20200007559A1

## CPC

- H04L 63/1416: {Event detection, e.g. attack signature detection}
- H04L 63/1483: {service impersonation, e.g. phishing, pharming or web spoofing (detection of rogue wireless access points H04W12/12)}
- G06F 16/951: Indexing; Web crawling techniques
- H04L 63/1425: {Traffic logging, e.g. anomaly detection}
- G06F 16/955: using information identifiers, e.g. uniform resource locators [URL]
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Examples of the present disclosure describe systems and methods for evaluating malicious web content for associated threats using specialized web crawling techniques. In aspects, a first set of malicious and/or potentially malicious resource identifiers is identified. The first set of resource identifiers is evaluated to determine at least a second set of resource identifiers associated with the first set of resource identifiers. The second set of resource identifiers are provide to a web crawling component, which scans the second set of resource identifiers using a threat detection component. If any resource identifiers in the second set of resource identifiers are identified as malicious (or potentially malicious), those resource identifiers may be classified and recorded, provided to the web crawling component, and/or added to the first set of resource identifiers in subsequent threat detection analyses.

## Claims

### Claim 1

A system comprising:
at least one processor; and
memory coupled to the at least one processor, the memory comprising computer executable instructions that, when executed by the at least one processor, performs a method comprising:
accessing a first set of resource identifiers;
determining a second set of resource identifiers associated with the first set of resource identifiers, wherein determining the second set of resource identifiers comprises evaluating one or more domains of the first set of resource identifiers;
evaluating the second set of resource identifiers to determine a third set of resource identifiers identified using the second set of resource identifiers; and
evaluating the third set of resource identifiers to determine whether one or more resource identifiers in the third set of resource identifiers is malicious.

### Claim 2 (depends on 1)

The system of claim 1, wherein the first set of resource identifiers comprises one or more resource identifiers identified as at least one of malicious and suspicious.

### Claim 3 (depends on 1)

The system of claim 1, wherein determining the second set of resource identifiers comprises investigating at least one of: root domains and sub-domains of the first set of resource identifiers, internal and external links associated with the first set of resource identifiers, one or more IP addresses hosting the first set of resource identifiers, a geolocation of one or more IP addresses associated with the first set of resource identifiers, and other domains owned by a resource.

### Claim 4 (depends on 3)

The system of claim 3, wherein determining the second set of resource identifiers further comprises at least one of: determining domain registration for the first set of resource identifiers, identifying common execution paths for web threats, and comparing web threat execution paths for various IP addresses associated with the first set of resource identifiers.

### Claim 5 (depends on 1)

The system of claim 1, wherein evaluating the second set of resource identifiers comprises providing the second set of resource identifiers to a web crawling utility.

### Claim 6 (depends on 5)

The system of claim 5, wherein the web crawling utility scans the second set of resource identifiers to identify one or more links embedded in the second set of resource identifiers.

### Claim 7 (depends on 1)

The system of claim 1, wherein evaluating the third set of resource identifiers comprises applying to the third set of resource identifiers at least one of a rule set, an evaluation model, and an algorithm.

### Claim 8 (depends on 1)

The system of claim 1, wherein evaluating the third set of resource identifiers comprises arranging the third set of resource identifiers into one or more classification groups, wherein the classification groups represent one or more web threat levels.

### Claim 9 (depends on 8)

The system of claim 8, wherein evaluating the third set of resource identifiers further comprises generating a set of metrics identifying at least one of: a number of resource identifiers evaluated, a number of unique resource identifiers evaluated, and a number of threats detected.

### Claim 10 (depends on 1)

The system of claim 1, wherein the method further comprises:
when one or more resource identifiers in the third set of resource identifiers is determined to be malicious, generating a set of instructions for performing one or more remedial actions based on the determination.

### Claim 11 (depends on 10)

The system of claim 10, wherein the one or more remedial actions comprise at least one of: displaying a warning, generating a report, blocking access to one or more resource identifiers, and initiating an anti-exploit utility.

### Claim 12

A method comprising:
determining a first set of resource identifiers;
determining a second set of resource identifiers associated with the first set of resource identifiers, wherein determining the second set of resource identifiers comprises evaluating one or more domains of the first set of resource identifiers;
evaluating the second set of resource identifiers to determine a third set of resource identifiers identified using the second set of resource identifiers; and
evaluating the third set of resource identifiers to determine whether one or more resource identifiers in the third set of resource identifiers is malicious.

### Claim 13 (depends on 12)

The method of claim 12, wherein the first set of resource identifiers comprises one or more resource identifiers previously identified as associated with malicious content.

### Claim 14 (depends on 12)

The method of claim 12, wherein determining the second set of resource identifiers further comprises evaluating at least one of: links associated with the first set of resource identifiers, an IP addresses hosting the first set of resource identifiers, one or more IP addresses associated with the IP addresses hosting the first set of resource identifiers, and a geolocation of one or more IP addresses.

### Claim 15 (depends on 14)

The method of claim 14, wherein the second set of resource identifiers is determined using an investigation utility for scanning resource identifiers.

### Claim 16 (depends on 12)

The method of claim 12, wherein evaluating the second set of resource identifiers comprises providing the second set of resource identifiers to a web crawling utility.

### Claim 17 (depends on 16)

The method of claim 16, wherein the web crawling utility is operable to scan the second set of resource identifiers to identify one or more links associated with the second set of resource identifiers.

### Claim 18 (depends on 12)

The method of claim 12, wherein evaluating the third set of resource identifiers comprises providing the third set of resource identifiers to a threat detection utility.

### Claim 19 (depends on 18)

The method of claim 18, wherein the threat detection utility is operable to determine whether one or more resource identifiers of the third set of resource identifiers is at least suspicious.

### Claim 20

A computer-readable media storing computer executable instructions that when executed cause a computing system to perform a method comprising:
determining a first set of resource identifiers;
determining a second set of resource identifiers associated with the first set of resource identifiers, wherein determining the second set of resource identifiers comprises evaluating one or more domains of the first set of resource identifiers;
evaluating the second set of resource identifiers to determine a third set of resource identifiers identified using the second set of resource identifiers; and
evaluating the third set of resource identifiers to determine whether one or more resource identifiers in the third set of resource identifiers is malicious.
