System and method for automated web site content analysis

A computer-implemented method for automated analysis of web source content, the method comprising:

• analyzing a web site structure to identify web page documents of the web site;
• for the web page documents:
• i. defining text fragments and their types based on content marking by content tags;
• ii. deriving terms from a dictionary;
• iii. finding the terms within the text fragments;
• iv. determining a suspiciousness parameter value of the text by counting a number of terms with a lexical property “suspicious” in the text fragment, wherein the suspiciousness parameter is “true” if a predetermined number of terms with the “suspicious” lexical property is found in the text;
• v. determining a genre of the web page document;
• vi. defining thematic properties of the text of the web page document based on the thematic characteristics of terms found in the text;
• vii. saving the genre and thematic properties of the text in a document index;
• viii. finding “child” web page documents linked to the web page documents that have the “suspicious” status and assigning the “suspicious” status to the “child” web page documents;
• ix. repeating analysis of the text of the web page document if the “suspicious” status is changed;
• determining a genre of the web site based on the genres of the web page documents;
• saving the site genre property into a site index;
• calculating a document disallowance parameter based on rules for all of the web site documents with the “suspicious” status and saving the disallowance parameters into web site document indices;
• assigning a “disallowed” status to the web page documents if its disallowance parameter exceeds a threshold;
• saving the “disallowed” status to the web page documents into a web page index;
• finding a document with a maximum value of the disallowance parameter in a web site index and saving the disallowance parameter as a site disallowance parameter;
• assigning the “disallowed” status to the web site if its disallowance parameter exceeds a threshold;
• and
• blocking the web site based on the “disallowed” status of the web site,
• wherein the rules include the thematic properties of the texts of the web page documents, genre properties of the web page documents and a genre property of the web site.