Configurable web crawler

A method for performing a user-configured web crawl, comprising:

• receiving a user-specified web crawl configuration for crawling a web to inspect text of resources accessible on the web, each resource identified by a unique uniform resource locator (URL), the user-specified web crawl configuration comprising user-specified crawling rules specifying crawling behavior,
• maintaining a queue of unprocessed URLs of resources;
• retrieving from the queue a next unprocessed URL;
• retrieving the crawling rules for the retrieved URL, the crawling rules specifying crawling behavior for at least one each of a general crawl behavior, one or more page-specific crawl behavior to be applied during a crawl to one or more identified pages, and one or more element-specific crawl behavior to be applied during a crawl to one or more identified page elements;
• if allowed by the specified general crawl behavior and page-specific crawl behavior of the retrieved crawling rules, fetching the page resource addressed by the retrieved URL;
• processing the fetched page resource on an element-by-element basis, by parsing a next page element from the fetched page resource, and if an element-specific behavior is specified in the retrieved crawling rules for the parsed next page element then operating with regard to the element according to the specified element-specific behavior, and otherwise if a page-specific behavior is specified for the fetched page resource then operating with regard to the element according to the specified page-specific behavior, and otherwise if a global crawl behavior is specified then operating with regard to the element according to the specified global crawl behavior, and otherwise operating with regard to the element according to a default crawl behavior;
• and repeating the processing until either all elements have been parsed and processed in accordance with the retrieved crawling rules, and wherein the default crawl behavior comprises extracting URLs of outbound links and adding the extracted URLs to the queue, storing text to a resource repository for further analysis;
• and
• repeating the steps of retrieving the next unprocessed URL through this repeating step until the queue is empty of unprocessed URLs;
• and
• wherein the retrieved crawling rules define the crawling behavior in terms of whether to record the resource text, whether to follow hyperlinks of the resource, whether to record text description in metadata associated with the page, whether to record keywords identified in the metadata associated with the page, whether to override a page title element, whether to exclude one or more identified resource pages, whether to exclude one or more identified domains.

A method for configuring a web crawl, the method comprising the steps of:

• specifying a starter seed uniform resource locator (URL) and a user-specified web crawl configuration to a configurable web crawler which crawls a web to inspect text of page resources accessible on the web, each page resource identified by a unique URL, the user-specified web crawl configuration comprising user-specified crawling rules specifying crawling behavior, the crawling rules specifying crawling behavior for at least one each of a general crawl behavior, one or more page-specific crawl behavior to be applied during a crawl to one or more identified pages, and one or more element-specific crawl behavior to be applied during a crawl to one or more identified page elements, wherein the received crawling rules define the crawling behavior in terms of whether to record the resource text, whether to follow hyperlinks of the resource, whether to record text description in metadata associated with the page, whether to record keywords identified in the metadata associated with the page, whether to override a page title element, whether to exclude one or more identified resource pages, whether to exclude one or more identified domains;
• wherein the configurable web crawler is configured to maintain a queue of unprocessed URLs of resources, insert the specified starter seed URL into the queue of unprocessed URLs of resources, retrieve from the queue a next unprocessed URL, retrieve the user-specified web crawl configuration which specifies the crawling rules, and if allowed by the specified general crawl behavior and page-specific crawl behavior of the retrieved crawling rules, fetch the page resource addressed by the retrieved URL and process the fetched page resource on an element-by-element basis by parsing a next page element from the fetched page resource, and if an element-specific behavior is specified in the retrieved crawling rules for the parsed next page element then operating with regard to the element according to the specified element-specific behavior, and otherwise if a page-specific behavior is specified for the fetched page resource then operating with regard to the element according to the specified page-specific behavior, and otherwise if a global crawl behavior is specified then operating with regard to the element according to the specified global crawl behavior, and otherwise operating with regard to the element according to a default crawl behavior;
• and repeating the processing until either all elements have been parsed and processed in accordance with the retrieved crawling rules, and wherein the default crawl behavior comprises extracting URLs of outbound links and adding the extracted URLs to the queue, storing text to a resource repository for further analysis, the one or more processing units repeating the processing on available unprocessed URLs in the queue.