Unsupervised, automated web host dynamicity detection, dead link detection and prerequisite page discovery for search indexed web pages

A volatile or non-volatile machine-readable medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform:

• retrieving, from a site host, pages associated with the site, wherein the pages contain content;
• determining how dynamic the content of the site is, based on the degree to which the content of the retrieved pages changed since a previous crawl of the site;
• if the content of the site is determined dynamic, in relation to a corresponding threshold, then continuing retrieving, from the site host, pages associated with the site;
• and
• if the content of the site is determined not dynamic, in relation to the corresponding threshold, then not retrieving, from the site host, a subset of pages associated with the domain;
• wherein determining how dynamic the content of the site is comprises maintaining a count of the number of continuous pages for which a same particular prerequisite page must be retrieved to retrieve the continuous pages;
• wherein retrieving a first page of the continuous pages without first retrieving the same prerequisite page causes a link to the first page to appear as a dead link but retrieving the first page after retrieving the same prerequisite page causes the link to the first page to appear as an alive link;
• wherein retrieving the same prerequisite page causes information required for retrieving the first page to be retrieved from the site host;
• wherein the method is performed by one or more computing devices comprising one or more processors.

A computer-implemented method for determining a prerequisite page for a page while checking, in an automated manner, page links associated with a site domain that was previously crawled, the method comprising:

• retrieving, from crawler storage, information that indicates whether or not retrieving the page requires setting request information by retrieving a prerequisite page, wherein the information identifies one or more ancestor pages that were retrieved prior to retrieving the page during a previous crawl and the order in which the one or more ancestor pages were retrieved, and wherein retrieving the prerequisite page causes information required for retrieving the page to be retrieved;
• if the page requires setting request information by retrieving a prerequisite page, then
• (a) retrieving a first ancestor page, wherein the first ancestor page is the ancestor page that was retrieved immediately prior to retrieving the page during the previous crawl;
• (b) after performing step (a), retrieving the page;
• (c) determining whether the page is dead or alive;
• (d) if the page is alive, then determining that the first ancestor page is a prerequisite page for the page;
• and
• (e) if the page is dead, then retrieving a second ancestor page, wherein the second ancestor page is the ancestor page that was retrieved immediately prior to retrieving the first ancestor page during the previous crawl, and repeating steps (b), (c) and (d) for the second ancestor page and, if necessary, repeating step (e) for the next ancestor page in the order;
• wherein the method is performed by one or more computing devices comprising one or more processors.

A computer-implemented method comprising:

• retrieving, from a site host, pages associated with the site, wherein the pages contain content;
• determining how dynamic the content of the site is, based on a count of the number of continuous pages for which a same prerequisite page must be retrieved to retrieve the continuous pages, wherein the count is reset upon retrieval of a different prerequisite page;
• if the content of the site is determined dynamic, in relation to a corresponding threshold, then continuing retrieving, from the site host, pages associated with the site;
• and
• if the content of the site is determined not dynamic, in relation to the corresponding threshold, then not retrieving, from the site host, a subset of pages associated with the domain;
• wherein retrieving a first page of the continuous pages without first retrieving the same prerequisite page causes a link to the first page to appear as a dead link but retrieving the first page after retrieving the same prerequisite page causes the link to the first page to appear as an alive link;
• wherein retrieving the same prerequisite page causes information required for retrieving the first page to be retrieved from the site host;
• wherein the method is performed by one or more computing devices comprising one or more processors.