# US 2018/0025012 A1 · Web page classification based on noise removal > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0025012 A1 - Document ID: US20180025012A1 - Status: Published application - Kind: A1 - Application number: 15/214,245 - Filed: 2016-07-19 - Published: 2018-01-25 - Earliest claimed: 2016-07-19 - Assignees: Fortinet, Inc. - Applicants: Fortinet, Inc. - Inventors: Xiping Cao; Ye Ma - Web page: https://patentgrep.com/patent/US20180025012A1 ## CPC - G06F 16/958: Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking - G06F 16/287: {Visualization; Browsing} - G06F 16/9558: {Details of hyperlinks; Management of linked annotations} - H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP] - H04L 63/0236: {Filtering by address, protocol, port number or service, e.g. IP-address or URL} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Systems and methods for improving accuracy of web content classification by removing perceived noise are provided. The system receives a Uniform Resource Locator (URL) of a web page that needs to be classified, and parses the web page so as to construct a tree containing a list of tags. Unwanted tags are removed from the list of tags to yield a tree containing only desired tags that form part of the web page. Subsequently, a list of hyperlinks are based on processing of the tree having desired tags, wherein the list of hyperlinks can include unwanted/undesired/invalid hyperlinks and valid hyperlinks. Unwanted hyperlinks can accordingly be removed from the list of hyperlinks, and each valid hyperlink can be categorized based on a list of categories, and a final category for the web page is determined based on a vector analysis of each category assigned to each valid hyperlink. ## Claims ### Claim 1 A system for web page classification comprising: a non-transitory storage device having embodied therein one or more routines operable to facilitate categorization of content of a web page; and one or more processors coupled to the non-transitory storage device and operable to execute the one or more routines, wherein the one or more routines include: a Uniform Resource Locator (URL) receive module, which when executed by the one or more processors, receives a URL of a web page to be categorized; a URL tree construction module, which when executed by the one or more processors, constructs a tree for the web page, wherein the tree represents a layout and a hierarchy of a plurality of tags that are used to represent the web page; a tag based filtration module, which when executed by the one or more processors, filters out a first set of tags from the plurality of tags to obtain desired tags that are indicative of relevant and actual content displayed by or linked by the web page; a hyperlink list retrieval module, which when executed by the one or more processors, retrieves a list of hyperlinks that form part of the web page based on processing of the desired tags; a valid hyperlink list generation module, which when executed by the one or more processors, processes the list of hyperlinks to generate a valid hyperlink list based on rejection of any or a combination of irrelevant hyperlinks, stop hyperlinks, and hyperlinks having a distance from a valid hyperlink of greater than a defined threshold; and a valid hyperlink list based categorization module, which when executed by the one or more processors, processes the valid hyperlink list to associate a final category from a plurality of categories with the web page. ### Claim 2 (depends on 1) The system of claim 1, wherein the first set of tags comprises tags indicative of display parameters of the web page, tags indicative of a template associated with the web page, tags indicative of layout parameters of the web page, tags indicative of advertisement information to be displayed concurrently with the content of the web page, leaf node tags, and tags indicative of formatting attributes of the web page. ### Claim 3 (depends on 1) The system of claim 1, wherein the valid hyperlink list based categorization module is further configured to, for each valid hyperlink in the valid hyperlink list, associate a category from the plurality of categories to the valid hyperlink, to generate a category vector containing information regarding a number of valid hyperlinks observed within the web page that are associated with the plurality of categories and to identify the final category based on the category from the plurality of categories whose number is greatest. ### Claim 4 (depends on 1) The system of claim 1, wherein the URL tree construction module is configured to preprocess the web page to construct the tree. ### Claim 5 (depends on 1) The system of claim 1, wherein the final category is associated with the URL. ### Claim 6 (depends on 1) The system of claim 1, wherein the final category is selected from any or a combination of News, Sports, Current affairs, Movies, Television, Entertainment, Business, Technology, Photos, Blogs, Country, World, City, Life & Style, Porn, Malicious URL, Phishing URL, Spamming URL, Malware URL, and a multi-type attack URL. ### Claim 7 (depends on 1) The system of claim 1, wherein the web page is assigned one or more sub-categories within the final category based on processing of the valid hyperlink list with respect to a list of available sub-categories within the final category. ### Claim 8 (depends on 1) The system of claim 1, wherein the web page is represented in a form of a HyperText Markup Language (HTML) or an extensible HyperText Markup Language (XHTML) document. ### Claim 9 (depends on 1) The system of claim 1, wherein one or more final categories are associated with the web page based on the processing of the valid hyperlink list. ### Claim 10 A method comprising: receiving, by a computer system, a Uniform Resource Locator (URL) of a web page to be categorized; constructing, by the computer system, a tree for the web page, wherein the tree represents a layout and a hierarchy of a plurality of tags that are used in the web page; filtering out, by the computer system, a first set of tags from the plurality of tags to obtain desired tags that are indicative of relevant and actual content displayed by or linked by the web page; retrieving, by the computer system, a list of hyperlinks that form part of the web page based on processing of the desired tags; processing, by the computer system, the list of hyperlinks to generate a valid hyperlink list based on rejection of any or a combination of irrelevant hyperlinks, stop hyperlinks, and hyperlinks having a distance from a valid hyperlink of greater than a defined threshold; and processing, by the computer system, the valid hyperlink list to associate a final category from a plurality of categories with the web page. ### Claim 11 (depends on 10) The method of claim 10, wherein the first set of tags comprise tags indicative of display parameters of the web page, tags indicative of a template of the web page, tags indicative of layout parameters of the web page, tags indicative of advertisement information to be displayed concurrently with content of the web page, leaf node tags, and tags indicative of formatting attributes of the web page. ### Claim 12 (depends on 10) The method of claim 10, further comprising for each valid hyperlink in the valid hyperlink list: associating a category from the plurality of categories to the valid hyperlink, to generate a category vector containing information regarding a number of valid hyperlinks observed within the web page that are associated with the plurality of categories; and identifying the final category based on the category of the plurality of categories whose number is greatest. ### Claim 13 (depends on 10) The method of claim 10, further comprising pre-processing the web page to construct the tree. ### Claim 14 (depends on 10) The method of claim 10, further comprising associating the final category with the URL. ### Claim 15 (depends on 10) The method of claim 10, wherein the final category is selected from any or a combination of News, Sports, Current affairs, Movies, Television, Entertainment, Business, Technology, Photos, Blogs, Country, World, City, Life & Style, Porn, Malicious URL, Phishing URL, Spamming URL, Malware URL, and a multi-type attack URL. ### Claim 16 (depends on 10) The method of claim 10, further comprising assigning the web page to one or more sub-categories within the final category based on processing of the valid hyperlink list with respect to a list of available sub-categories within the final category. ### Claim 17 (depends on 10) The method of claim 10, wherein the web page comprises a HyperText Markup Language (HTML) or an extensible HyperText Markup Language (XHTML) document. ### Claim 18 (depends on 10) The method of claim 10, wherein one or more final categories are associated with the web page based on the processing of the valid hyperlink list.