Back to results
US 2012/0259833 A1Published application
Configurable web crawler
AssigneeVistaprint Technologies Limited
InventorAndrei Paduroiu
Life of this patent
Estimating the term…
Abstract
A configurable web crawler allows a user configure a web crawl by specifying one or more of thread throttling rules, domain restriction rules, and crawling rules. The configurable web crawler crawls the web beginning with a seed uniform resource locator and according to the crawl configuration.
Claims33
A configurable web crawler system, comprising:
- one or more processors configured to allow a user to configure a web crawl according to one or more of thread throttling rules, domain restriction rules, and crawling rules;
- a crawling function which receives a starter seed uniform resource locator, and crawls a web, beginning at the starter seed uniform locator, according to the configured web crawl.
A method for configuring a web crawl, comprising:
- receiving by one or more processors a starter seed uniform resource locator and a web crawl configuration comprising one or more of thread throttling rules, domain restriction rules, and crawling rules;
- and
- crawling a web, beginning at the starter seed uniform locator, according to the web crawl configuration.
A method for configuring a web crawl, the method comprising the steps of:
- specifying to a configurable web crawler a starter seed uniform resource locator and a web crawl configuration comprising one or more of thread throttling rules, domain restriction rules, and crawling rules;
- wherein the configurable web crawler is configured to crawl a web beginning at the starter seed uniform locator according to the web crawl configuration.
Dependencies are inferred from the claim text.