Techniques in using feedback in crawling web content
AssigneeYahoo! Inc.
InventorsAmit Jaiswal, Ravikiran Meka, Binu Raj
Life of this patent
Estimating the term…
Abstract
A method for providing feedback to a web crawler is provided. A content processor determines whether a crawled web page is useful to an application. This determination is passed to a learning module. The learning module analyzes crawled web pages and the determinations of usefulness made by the content processor and generates rules for crawling more useful web pages and less non-useful web pages. The learning module provides these rules to the crawler, which applies them in making crawling decisions. Rules expire after a period of time. Paths from a web site's main web page to useful web pages are preserved. A token tree is constructed to facilitate the generation of rules.
Claims20
A method for using feedback to improve web-crawling, comprising:
- receiving, from a web-crawler module, crawled information, wherein the crawled information comprises any combination of URL, contents, attributes, and meta-information of a crawled webpage,
- wherein the meta-information comprises inlinks and outlinks of the crawled webpage;
- evaluating, based on the crawled information, whether the crawled web page is useful to an application;
- generating, based on the crawled information and whether the crawled web page is useful, at least one rule for choosing webpages for crawling;
- sending the at least one rule to the web-crawler module;
- at the web-crawler module, performing the steps of:
- choosing a next webpage based at least in part on the at least one rule;
- and
- crawling the next webpage.
A computer-implemented computer-readable storage medium storing instructions for using feedback to improve web-crawling, wherein the instructions include instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of:
- receiving, from a web-crawler module, crawled information, wherein the crawled information comprises any combination of URL, contents, attributes, and meta-information of a crawled webpage,
- wherein the meta-information comprises inlinks and outlinks of the crawled webpage;
- evaluating, based on the crawled information, whether the crawled web page is useful to an application;
- generating, based on the crawled information and whether the crawled web page is useful, at least one rule for choosing webpages for crawling;
- sending the at least one rule to the web-crawler module;
- at the web-crawler module, performing the steps of:
- choosing a next webpage based at least in part on the at least one rule;
- and
- crawling the next webpage.
Dependencies are inferred from the claim text.