# US 10,122,720 B2 · System and method for automated web site content analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,122,720 B2
- Document ID: US10122720B2
- Status: Granted
- Kind: B2
- Application number: 15/426,763
- Filed: 2017-02-07
- Published: 2018-08-09
- Granted: 2018-11-06
- Earliest claimed: 2017-02-07
- Assignees: Plesk International GmbH
- Applicants: Parallels International GmbH
- Inventors: Sergey Oleynikov; Yury Zagorulko; Elena Sidorova
- Web page: https://patentgrep.com/patent/US10122720B2

## CPC

- H04L 63/10: {for controlling access to devices or network resources}
- G06F 11/302: {where the computing system component is a software system}
- G06F 11/3055: {Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available (error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction of the data by redundancy in operations G06F11/14; error detection or correction by redundancy in hardware G06F11/16)}
- G06F 16/3344: {using natural language analysis}
- G06F 16/353: into predefined classes
- G06F 16/986: {Document structures and storage, e.g. HTML extensions}
- G06F 21/6218: {to a system of files or objects, e.g. local or distributed file system or database}
- G06F 40/279: Recognition of textual entities
- G06F 40/30: Semantic analysis
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06F 2201/81: Threshold
- G06F 2201/875: Monitoring of systems including the internet
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system and method for an automated web source content analysis. The system of automated content analysis performs the following: a search of terms, i.e. key words and phrases, presented in the special dictionary, in the text content; executes a multi-factor genre content analysis based on structural, pragmatic and stylistics properties; executes thematic content analysis using a rubricator built based on illegal subjects and topics and their antagonists; and the system makes a decision based on a combination of thematic and genre properties of the text. The proposed method allows for providing a final decision in terms that are easily understood by a user.

## Claims

### Claim 1

A computer-implemented method for automated analysis of web source content, the method comprising:
analyzing a web site structure to identify web page documents of the web site;
for the web page documents:
i. defining text fragments and their types based on content marking by content tags;
ii. deriving terms from a dictionary;
iii. finding the terms within the text fragments;
iv. determining a suspiciousness parameter value of the text by counting a number of terms with a lexical property “suspicious” in the text fragment, wherein the suspiciousness parameter is “true” if a predetermined number of terms with the “suspicious” lexical property is found in the text;
v. determining a genre of the web page document;
vi. defining thematic properties of the text of the web page document based on the thematic characteristics of terms found in the text;
vii. saving the genre and thematic properties of the text in a document index;
viii. finding “child” web page documents linked to the web page documents that have the “suspicious” status and assigning the “suspicious” status to the “child” web page documents;
ix. repeating analysis of the text of the web page document if the “suspicious” status is changed;
determining a genre of the web site based on the genres of the web page documents;
saving the site genre property into a site index;
calculating a document disallowance parameter based on rules for all of the web site documents with the “suspicious” status and saving the disallowance parameters into web site document indices;
assigning a “disallowed” status to the web page documents if its disallowance parameter exceeds a threshold;
saving the “disallowed” status to the web page documents into a web page index;
finding a document with a maximum value of the disallowance parameter in a web site index and saving the disallowance parameter as a site disallowance parameter;
assigning the “disallowed” status to the web site if its disallowance parameter exceeds a threshold; and
blocking the web site based on the “disallowed” status of the web site,
wherein the rules include the thematic properties of the texts of the web page documents, genre properties of the web page documents and a genre property of the web site.

### Claim 2 (depends on 1)

The method of claim 1, wherein the lexical properties of the terms is a vector with Boolean values indicating whether the terms belong to any of the following groups of terms:
professional terms;
social group terms;
slang terms;
narcotics-related terms;
terrorism-related terms;
obscene language; and
suspicious terms.

### Claim 3 (depends on 1)

The method of claim 1, wherein the web page document genre reflects a purpose and an activity provided by the web page.

### Claim 4 (depends on 1)

The method of claim 1, further comprising accounting of the genre in an analyzing algorithm and forming a training set.

### Claim 5 (depends on 1)

The method of claim 1, further comprising assigning the “disallowed” marker based on the vectors of relevancies of the text content of web documents to topics from the thematic rubricator, on the vectors of relevancies of the text content of web documents to genres of documents from a genre rubricator, and on the vector of relevancy of the entire web site content to genres of the web site from the genre rubricator.

### Claim 6 (depends on 1)

The method of claim 1, wherein the thematic properties of the text represents a vector of relevance of the text to a set of topics ti from a thematic rubricator, the vector of relevance defined as Pt=(p(t1), p(t2), . . . , p(ti), . . . , p(tNt)), where p(ti) is the relevance of the text to topic ti, i∈[1, . . . , Nt], where Nt is a number of topics in the thematic rubricator.

### Claim 7 (depends on 6)

The method of claim 6, wherein for each occurrence of a coincident implementation of a topic ti, a genre jk, a site genre jsm and a lexical characteristic lexq, a corresponding disallowing marker m− or allowing marker m+ is assigned.

### Claim 8 (depends on 7)

The method of claim 7, further calculating of probabilities of an “allowed” combination of a topic ti, a genre jk, a site genre jsm and a lexical characteristic lexq, and “disallowed” one, as follows:
M−=Σp(ti)·p(jk)·p(jsm)·lexq, such that topic ti, a genre jk, a site genre jsm and a lexical characteristic lexq correspond to disallowing marker m−; and
M+=Σp(ti)·p(jk)·p(jsm)·lexq, such that topic ti, a genre jk, a site genre jsm and a lexical characteristic lexq correspond to allowing marker m+.

### Claim 9 (depends on 8)

The method of claim 8, further comprising normalizing the probabilities so that a sum of all the probabilities is 1.

### Claim 10 (depends on 8)

The method of claim 8, further comprising making a final decision about blocking or allowing the web document content based on a C criterion, wherein
C=(M−−M+).

### Claim 11 (depends on 10)

The method of claim 10, further comprising disallowing the web site content, if C>0 for at least one document.

### Claim 12 (depends on 1)

The method of claim 1, wherein the web page document is any of:
a structured file containing text and images;
a structured text file;
an HTML/XML file;
a JSON file; and
a plain text file.

### Claim 13 (depends on 1)

The method of claim 1, further comprising saving the suspiciousness parameter value into the web page document index as the web page document having a “suspicious” status.

### Claim 14 (depends on 1)

The method of claim 1, wherein the determining of the genre of the web page document is based on comparing genre markers of the web page document to genre markers from the dictionary.

### Claim 15 (depends on 1)

The method of claim 1, wherein the thematic properties of the text are calculated based on thematic characteristics of the terms in the text.

### Claim 16 (depends on 1)

A non-transitory computer storage medium storing computer code for executing the steps of claim 1.

### Claim 17

A system for automated analysis of web source content, the system comprising:
for web page documents of a website, computer program code for:
i. defining text fragments and their types based on content marking by content tags;
ii. deriving terms from a dictionary;
iii. finding the terms within the text fragments;
iv. determining a suspiciousness parameter value of the text by counting a number of terms with a lexical property “suspicious” in the text fragment, wherein the suspiciousness parameter is “true” if a predetermined number of terms with the “suspicious” lexical property is found in the text;
v. defining thematic properties of the text of the web page document based on the thematic characteristics of terms found in the text;
vi. saving genre and thematic properties of the text in a document index;
vii. assigning a “suspicious” status to any “child” web page documents;
viii. repeating analysis of the text of the web page document if the “suspicious” status is changed;
computer program code for
a. determining a genre of the web site based on the genres of the web page documents;
b. saving the genre of the website into a site index;
c. calculating a document disallowance parameter based on rules for all of the web site documents with the “suspicious” status and saving the disallowance parameters into the web site document index;
d. assigning a “disallowed” status to the web page documents if its disallowance parameter exceeds a threshold;
e. saving the “disallowed” status of the web page documents into a web page index;
f. finding a document with a maximum value of the disallowance parameter in the web site index and saving the disallowance parameter as a site disallowance parameter;
g. assigning the “disallowed” status to the web site if its disallowance parameter exceeds a threshold; and
h. blocking the web site based on the “disallowed” status of the web site,
wherein the rules include the thematic properties of the texts of the web page documents, genre properties of the web page documents and a genre property of the web site.