Method and system for triggering web crawling based on registry data
AssigneeVeriSign, Inc.
InventorsKarthik Shyamsunder, Srinivas Sunkara, Alexander Durych Holmes, Andrew Simpson, Phani Pattappu
Life of this patent
Estimating the term…
Abstract
A method of triggering crawling of a domain includes receiving information related to a domain from a registrar and processing the information related to the domain. The method also includes storing the processed information in a registry zone file and forming a list of registry data based on the processed information. The list of registry data comprises a subset of the registry zone file. The method further includes crawling one or more of the domains in the list of registry data.
Claims24
A method of triggering crawling of a domain, the method comprising:
- receiving information related to a domain from a registrar;
- processing the information related to the domain;
- storing the processed information in a registry zone file;
- forming a list of registry data based on the processed information, wherein the list of registry data comprises a subset of the registry zone file;
- and
- crawling one or more of the domains in the list of registry data.
A method of performing a web site crawl, the method comprising:
- receiving a zone file update report comprising a list of domains associated with changes in registration status during a predetermined time period;
- and
- crawling one or more web sites associated with the domains listed in the zone file update report.
A method of providing domain information to a web crawler, the method comprising:
- receiving information related to a domain from a registrar during a predetermined time period;
- processing the information related to the domain;
- storing the processed information in a registry zone file;
- forming a list of registry data based on the processed information;
- establishing a threshold value for domain resolution requests;
- determining that a number of resolution requests for a second domain in the predetermined time period is greater than the threshold value;
- adding information related to the second domain to the list of registry data;
- and
- making the list of registry data accessible to a web crawler.
Dependencies are inferred from the claim text.