Networked architecture for enabling automated gathering of information from Web servers

A data-gathering and reporting system for collecting WEB summaries from the Internet for individual subscribers to a Portal subscription system, comprising:

• a plurality of gatherer servers each connected to the Internet, to an ascending hierarchy of work request distribution servers, and to a ascending hierarchy of collector servers;
• a work request generator at the top of the hierarchy of distribution servers, generating work requests for collecting WEB summaries;
• and
• a filer server at the top of the hierarchy of collector servers, the file server connected to and writing data to a database;
• wherein flow is by work requests from the work request generator down the hierarchy of distributor servers to the gatherer servers where work requests are accomplished by gathering WEB summaries from Internet servers according to the work requests, and by data collected from the gatherer servers up the hierarchy of collector servers to the filing server, and wherein gating of work assignments and data between one server and another in the hierarchy of distributor servers is by the one server having a queue with an adjustable threshold, and demanding data or work assignments from the other server as a result of the queue level falling to the threshold.