Efficient freshness crawl scheduling

A method for seeking updated copies of content items in a content set by following a crawl probability vector, the method comprising:

• receiving a bandwidth constraint that limits a total amount of crawls allowed to be made across the content set per unit time, wherein the content set comprises a first subset of content items with incomplete change information and a second subset of content items with complete change information;
• receiving a set of importance scores that comprises an individual importance score for each content item in the content set;
• receiving a set of change rates that comprises an individual change rate for each content item in the content set;
• determining an optimized split of bandwidth between the first subset and the second subset to produce a first bandwidth allocation to the first subset and a second bandwidth allocation to the second subset by minimizing a cost function that associated with the first subset and the second subset;
• outputting the first bandwidth allocation for use calculating a crawl rate schedule for the first subset and the second bandwidth allocation for use calculating a crawl probability vector for the second subset;
• calculating, using the second cost function, the crawl probability vector for the second subset that comprises an individual crawl rate for each individual content item in the second subset, wherein the set of change rates and the set of importance scores are input to the second cost function, wherein the crawl probability vector is a solution to the second cost function that minimizes a sum of costs across all content items in the second subset and is constrained by the second bandwidth allocation, such that a sum of crawl rates for all content items in the second subset is equal to the second bandwidth allocation;
• and
• retrieving updated copies of content items in the first subset according to the crawl probability vector and the second subset according to the crawl probability vector.

One or more computer storage media that, when executed by a computing device, causes the computing device to perform a method of seeking updated copies of content items in a content set, the method comprising:

• receiving a bandwidth constraint that limits a total amount of crawls the crawler is allowed to make across the content set per unit of time, wherein the content set comprises a first subset of content items with incomplete change information and a second subset of content items with complete change information;
• receiving a set of importance scores that comprises an individual importance score for each content item in the content set;
• receiving a set of change rates that comprises an individual change rate for each content item in the content set;
• determining an optimized split of bandwidth between the first subset and the second subset to produce a first bandwidth allocation to the first subset and a second bandwidth allocation to the second subset by minimizing a cost function associated with the first subset and the second subset;
• outputting the first bandwidth allocation for use calculating a crawl rate schedule for the first subset and the second bandwidth allocation for use calculating a crawl probability vector for the second subset;
• and
• retrieving updated copies of content items in the first subset according to the crawl rate schedule and the second subset according to the crawl probability vector.