Back to results
US 11,599,595 B2Granted
Determining whether to process identified uniform resource locators
AssigneeInternational Business Machines Corporation
InventorTimothy F. Jones
Life of this patent
Estimating the term…
Abstract
In one general embodiment, a computer-implemented method includes selecting a portion of a hash table, identifying a plurality of uniform resource locators (URLs) stored within the selected portion of the hash table, and for each of the plurality of identified URLs, individually determining whether to process the identified URL, based on data associated with the identified URL.
Claims20
A computer-implemented method, comprising:
- selecting one of a plurality of buckets within a hash table to be reviewed;
- identifying a plurality of uniform resource locators (URLs) stored within the selected bucket of the hash table;
- for each of the identified URLs, conditionally processing the identified URL, based on data associated with the identified URL, the processing including downloading a web page located at the URL,
- wherein conditionally processing the identified URL includes determining an overall score for the identified URL, based on global information associated with a plurality of known URLs in addition to metadata for the identified URL;
- and
- comparing the overall score to a threshold score,
- wherein the threshold score is an average overall score for all URLs within the hash table.
4 more elements
A computer-implemented method, comprising:
- selecting one of a plurality of buckets within a hash table to be reviewed;
- identifying a plurality of uniform resource locators (URLs) stored within the selected bucket of the hash table;
- and
- for each of the plurality of identified URLs, conditionally processing the identified URL, based on data associated with the identified URL,
- wherein conditionally processing the identified URL includes:
- determining an overall score for the identified URL, based on global information associated with a full set of known URLs in addition to metadata for the identified URL, wherein:
- the global information includes a time since a last crawl, a total number of pages crawled within the hash table, a total number of pages not crawled within the hash table, and a percentage of a global crawl goal that is not currently met, and
- the metadata for the identified URL includes a page score for the URL based on a link authority for the URL, a last crawl time for the URL, and changes made to a page associated with the URL since a last crawl,
- comparing the overall score to a threshold score, where the threshold score is an average overall score for all URLs within the hash table, and
- processing the URL in response to determining that the overall score exceeds the threshold score.
7 more elements
A computer-implemented method, comprising:
- selecting one of a plurality of buckets within a hash table to be reviewed;
- identifying a plurality of uniform resource locators (URLs) stored within the selected bucket of the hash table;
- and
- for each of the plurality of identified URLs, conditionally processing the identified URL, based on data associated with the identified URL,
- wherein:
- each bucket represents a range of URLs, and each URL is represented once within the hash table,
- each of the URLs is stored within the bucket as a digest,
- each of the URLs is transformed into a sequence of bytes represented by the digest utilizing a hash function,
- the data associated with each of the URLs is linked to its associated digest and is stored with the digest within the hash table,
- the data associated with each of the URLs includes metadata including:
- a page score for the URL based on a link authority for the URL, a last crawl time for the URL, and changes made to a page associated with the URL since a last crawl;
- and wherein conditionally processing the identified URL includes:
- determining an overall score for the identified URL, based on global information associated with a full set of known URLs in addition to the metadata for the identified URL, the global information including a time since a last crawl, a total number of pages crawled within the hash table, a total number of pages not crawled within the hash table, and a percentage of a global crawl goal that is not currently met,
- comparing the overall score to a threshold score, where the threshold score is an average overall score for all URLs within the hash table, and
- processing the URL in response to determining that the overall score exceeds the threshold score.
12 more elements
Dependencies are inferred from the claim text.