System and method for enhancing crawling by extracting requests for webpages in an information flow

A method for searching and analysing the traffic content at access points in data networks, wherein the data networks are shared network resources on the Internet, wherein the method comprises:

• collecting information in the form of data extracted from the information flow at said access points in the data network,
• indexing said collected information,
• searching said indexed information, and
• retrieving information based on the searching,
• wherein the searching step is implemented by at least one collaborating search engine, the searching step including substeps for dispatching search requests to said at least one collaborating search engine, collecting search result from a local traffic index, collecting search results from said at least one collaborating search engine and combining said collected search results to provide a unified result to an initial search request.

A method for searching and analysing the traffic content at access points in data networks, wherein the data networks are shared network resources on the Internet, wherein the method comprises:

• collecting information in the form of data extracted from the information flow at said access points in the data network,
• indexing said collected information,
• searching said indexed information,
• retrieving information based on the searching, and
• collecting document identifiers for requested documents, annotating said documents identifiers with spatial information about users submitting the requests, computing access statistics for at least one document including at least the number of document requests from a spatial region and the total number of requests from said spatial region, and determining which documents are most specific for a given spatial region by comparing the access statistics for said given spatial region with the corresponding access statistics for at least a second spatial region.

A method for searching and analysing the traffic content at access points in data networks, wherein the data networks are shared network resources on the Internet, wherein the method comprises:

• collecting information in the form of data extracted from the information flow at said access points in the data network,
• indexing said collected information,
• searching said indexed information, and
• retrieving information based on the searching,
• collecting document identifiers for the requested documents, annotating document requests such that consecutive requests from the same user can be identified, and computing a document similarity between a document “b” and a reference document “a” by comparing the number of “b” requests in the proximity of “a” requests with an average frequency of “b” requests.