Collaborative team crawling:Large scale information gathering over the internet

A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform a method for gathering objects from an information system, said method comprising:

• constructing logical web-graphs defining logical relationships between specified objects in the information system;
• partitioning the web-graph into sub-graphs;
• and
• processing the specified objects to locate desired objects by:
• balancing the processing of the specified objects by mapping the sub-graphs to one or more preferred processors, wherein a processor is preferred if it reduces processing time or overhead;
• and
• re-balancing the processing if the processing load becomes unbalanced.

An apparatus for traversing an information system to retrieve designated objects, the apparatus comprising:

• a recognizer to determine a format for each object retrieved;
• a summarizer to read varying data formats communicatively coupled to the recognizer;
• an expander for expanding objects having a compressed format communicatively coupled to the recognizer and the summarizer;
• storage and processors included in one or more of the above, the processors capable of reading digital signals to traverse the information system to retrieve designated objects by:
• constructing logical web-graphs defining logical relationships between specified objects in the information system;
• partitioning the web-graph into sub-graphs;
• and
• processing the specified objects to locate desired objects by:
• balancing the processing of the specified objects by mapping the sub-graphs to one or more preferred processors, wherein a processor is preferred if it reduces processing time or overhead;
• and
• re-balancing the processing if the processing load becomes unbalanced.

An apparatus for traversing an information system to retrieve designated objects, the apparatus comprising:

• a means for recognizing a format for an object retrieved;
• a means for summarizing and reading varying data formats communicatively coupled to the recognizing means;
• a means for expanding objects having a compressed format and communicatively coupled to the recognizing means and the summarizing means;
• storage means and processor means included in one or more of the above, the storage means for storing digital signals, the processor means for reading digital signals and traversing the information system to retrieve designated objects by:
• constructing logical web-graphs defining logical relationships between specified objects in the information system;
• partitioning the web-graph into sub-graphs;
• and
• processing the specified objects to locate desired objects by:
• balancing the processing of the specified objects by mapping the sub-graphs to one or more preferred processors, wherein a processor is preferred if it reduces processing time or overhead;
• and
• re-balancing the processing if the processing load becomes unbalanced.