Method and apparatus for web crawler data collection

A computer-implemented method for Web crawler data collection to assist a search engine in deciding on the optimal crawling frequencies and the optimal times to crawl each Web page, comprising the steps of:

• collecting information associated with a plurality of queries, wherein the information relates to results of the queries or-responses to the queries or both results and responses to the queries;
• using the collected information to determine a clicking probability cj,k relating to a frequency with which a client selects a returned Web page i in a particular position j of a particular query result k, wherein the clicking probability cj,k is determined by estimating a temporary clicking probability Tcj,k and then updating an existing clicking probability cj,k using a smoothing process;
• and
• using the collected information to determine an incorrect response probability di relating to the probability that a query to a stale version of a particular Web page i yields an incorrect or vacuous response, wherein determining an incorrect response probability di comprises:
• determining that a Previously Presented Web page i is in a sample queue that has not been processed;
• reading an index value i of the Web page i and determining if the Web page i exists;
• when the Web page exists, determining if the Web page is current;
• when the Web page i is not current, reading a client query from the sample queue and determining if the Web page i contains a correct response to the client query;
• and
• when the Web page i does not contain a correct response to the client query, updating di=D·di+(1−D), where D is a smoothing parameter.

A computer-implemented method for Web crawler data collection to assist a search engine in deciding on the optimal crawling frequencies and the optimal times to crawl each Web page, comprising the steps of:

• collecting information associated with a plurality of queries, wherein the information relates to results of the queries or responses to the queries or both results of and responses to the queries;
• and
• using the collected information to determine to determine an incorrect response probability di relating to the probability that a query to a stale version of a particular Web page i yields an incorrect or vacuous response, wherein the incorrect response probability is determined by:
• determining that a Previously Presented Web page i is in a sample queue that has not been processed;
• reading an index value i of the Web page i and determining if the Web page i exists;
• when the Web page exists, determining if the Web page is current;
• when the Web page i is not current, reading a client query from the sample queue and determining if the Web page i contains a correct response to the client query;
• and
• when the Web page i does not contain a correct response to the client query, updating di=D·di+(1−D), where D is a smoothing parameter.