Incremental crawling of multiple content providers using aggregation

A system for incremental crawling of content stored on a plurality of content providers using aggregation comprising:

• at least one processor configured to:
• receive a request to crawl content on one or more associated content providers, wherein the request comprises at least a starting index value and a range value associated with a quantity of content to be accessed on the associated content providers;
• forward the request to a first content provider on a list, in response to determining that there is no valid state information, wherein forwarding the request to a first content provider on a list further comprises:
• incrementing the range value by the starting index value;
• and
• passing timing information corresponding to the first content provider with the request;
• forward the request to a first content provider identified by the state information as a next content provider, in response to determining that there is valid state information based on a comparison of the starting index value with a last received index, wherein forwarding the request to a first content provider identified by the state information further comprises:
• setting the starting index value to a value indicated by the state information as a next starting index;
• and
• passing the state information and timing information corresponding to the first content provider with the request;
• receive references, state information, or timing information from the first content provider;
• aggregate the received references, state information, and timing information with other references, state information, and timing information, respectively;
• forward the request to a second content provider on the list, in response to a quantity of content from the first content provider being less than the quantity of content to be accessed on the associated content providers;
• update the state information with the next content provider and corresponding next starting index, in response to determining that the request includes no unsatisfied references that other content providers on the list are able to satisfy;
• and
• return the aggregated references, the updated state information, and the aggregated timestamp.

A computer program product for incremental crawling of content stored on a plurality of content providers using aggregation comprising:

• a non-transitory computer readable medium having computer readable program code embodied therewith, the computer readable program code comprising computer readable program code configured to:
• receive a request to crawl content on one or more associated content providers, wherein the request comprises at least a starting index value and a range value associated with a quantity of content to be accessed on the associated content providers;
• forward the request to a first content provider on a list, in response to determining that there is no valid state information, wherein forwarding the request to a first content provider on a list further comprises:
• incrementing the range value by the starting index value;
• and
• passing timing information corresponding to the first content provider with the request;
• forward the request to a first content provider identified by the state information as a next content provider, in response to determining that there is valid state information based on a comparison of the starting index value with a last received index, wherein forwarding the request to a first content provider identified by the state information further comprises:
• setting the starting index value to a value indicated by the state information as a next starting index;
• and
• passing the state information and timing information corresponding to the first content provider with the request;
• receive references, state information, or timing information from the first content provider;
• aggregate the received references, state information, and timing information with other references, state information, and timing information, respectively;
• forward the request to a second content provider on the list, in response to a quantity of content from the first content provider being less than the quantity of content to be accessed on the associated content providers;
• update the state information with the next content provider and corresponding next starting index, in response to determining that the request includes no unsatisfied references that other content providers on the list are able to satisfy;
• and
• return the aggregated references, the updated state information, and the aggregated timestamp.