System and method for crawl policy management utilizing ip address and ip address range

A method for configuring a policy management protocol for a web crawler, the method comprising:

• determining a web space that is to be crawled by a web crawler, wherein the web space is comprised of at least one IP address and at least one range of IP addresses;
• storing the at least one IP address and the at least one range of IP addresses within a URL frontier;
• determining additional hostnames that are associated with the at least one IP address by performing a reverse DNS lookup of the at least one IP address;
• configuring the web crawler to crawl the at least one IP address or at least one range of IP addresses, and determine additional hostnames that are associated with the at least one IP address or at least one range of IP addresses;
• and
• performing a web crawling function upon the determined the at least one IP address, the at least one range of IP addresses, and the determined additional hostnames by the web crawler;
• wherein determining the web space further comprises determining at least one IP address and one range of IP addresses that the web crawler is restricted to crawling;
• wherein the IP address and range of IP addresses are associated with geographic locations;
• wherein the IP address and range of IP addresses are associated with a predetermined taxonomy as defined by a regional Internet registry.