US 8,712,992 B2Granted

Method and apparatus for web crawling

AssigneeMicrosoft Corporation

InventorsAlexey Maykov, Matthew F. Hurst

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method and system for retrieving data from a webpage is described herein. A scheduler organizes, or rather orders, a group of webpage identifiers according to some predetermined criteria. Based upon this ordering, a fetcher may be configured to fetch data from webpages identified by the identifiers. To promote efficiency and reduce the latency between when a webpage is updated and when the fetcher retrieves data from the webpage, the scheduler may be configured to reorder the identifiers in such a manner that it causes an identifier that was less relevant, and would not have been sent to the fetcher, to become more relevant. In this way, the method and system may be particularly useful for retrieving data related to webpages that are updated frequently, such as social media webpages, for example.

Claims20

Dependent claims

A method, comprising:

receiving a set of webpage identifiers, a first webpage identifier of the set identifying a first webpage, a second webpage identifier of the set identifying a second webpage, and a third webpage identifier of the set identifying a third webpage;
performing an ordering of webpage identifiers of the set to derive a first order for crawling the first webpage, the second webpage, and the third webpage, the first order providing for crawling the first webpage before crawling the second webpage and providing for crawling the second webpage before crawling the third webpage, the ordering based at least in part upon:
viewership information pertaining to at least one of the first webpage, the second webpage, or the third webpage;
ping information pertaining to a ping received from at least one of the first webpage, the second webpage, or the third webpage;
and
modification frequency information pertaining to at least one of the first webpage, the second webpage, or the third webpage;
determining, while crawling the first webpage, that the first webpage and the third webpage are associated with a common host and that the second webpage is not associated with the common host, the determining comprising searching the set, while crawling the first webpage, for one or more webpage identifiers that are associated with the common host, the common host based upon an internet protocol address;
and
performing, while crawling the first webpage, a first reordering of the webpage identifiers of the set based at least in part upon the determining to derive a second order, the second order providing for crawling the first webpage and the third webpage at least one of concurrently or contiguously and before crawling the second webpage, at least some of the method implemented at least in part via a processing unit.

6 more elements

A computer readable device comprising computer executable instructions that when executed perform a method, comprising:

receiving a set of webpage identifiers, a first webpage identifier of the set identifying a first webpage, a second webpage identifier of the set identifying a second webpage, and a third webpage identifier of the set identifying a third webpage;
performing an ordering of webpage identifiers of the set to derive a first order for crawling the first webpage, the second webpage, and the third webpage, the first order providing for crawling the first webpage before crawling the second webpage and providing for crawling the second webpage before crawling the third webpage, the ordering based at least in part upon at least one of:
viewership information pertaining to at least one of the first webpage, the second webpage, or the third webpage;
ping information pertaining to a ping received from at least one of the first webpage, the second webpage, or the third webpage;
or
modification frequency information pertaining to at least one of the first webpage, the second webpage, or the third webpage;
determining, while crawling the first webpage, that the first webpage and the third webpage are associated with a common host, the determining comprising searching the set, while crawling the first webpage, for one or more webpage identifiers that are associated with the common host, the common host based upon an internet protocol address;
and
performing, while crawling the first webpage, a first reordering of the webpage identifiers of the set based at least in part upon the determining to derive a second order, the second order providing for crawling the first webpage and the third webpage at least one of concurrently or contiguously and before crawling the second webpage.

6 more elements

A system for crawling webpages, comprising:

one or more processing units;
and
memory configured to store instructions that when executed by at least one of the one or more processing units perform a method, comprising:
receiving a set of webpage identifiers, a first webpage identifier of the set identifying a first webpage, a second webpage identifier of the set identifying a second webpage, and a third webpage identifier of the set identifying a third webpage;
performing an ordering of webpage identifiers of the set to derive a first order for crawling the first webpage, the second webpage, and the third webpage, the first order providing for crawling the first webpage before crawling the second webpage and providing for crawling the second webpage before crawling the third webpage, the ordering based at least in part upon at least two of:
viewership information pertaining to at least one of the first webpage, the second webpage, or the third webpage;
ping information pertaining to a ping received from at least one of the first webpage, the second webpage, or the third webpage;
or
modification frequency information pertaining to at least one of the first webpage, the second webpage, or the third webpage;
determining, while crawling the first webpage, that the first webpage and the third webpage are associated with a common host, the common host based upon an internet protocol address;
and
performing, while crawling the first webpage, a first reordering of the webpage identifiers of the set based at least in part upon the determining to derive a second order, the second order providing for crawling the first webpage and the third webpage at least one of concurrently or contiguously and before crawling the second webpage, at least some of the method implemented at least in part via a processing unit.

9 more elements

Dependencies are inferred from the claim text.