Automatic tagging of documents and exclusion by content

A computer-based method for processing data retrieved during a crawl of a computer network, comprising:

• retrieving a document with a gatherer process;
• and
• parsing the document into a document data stream including contents and properties;
• and
• piping the document data stream to at least one active plug-in, said active plug-in having the capability of modifying the document data stream by adding to, deleting from, or changing the content and/or properties of the document data stream;
• and
• piping the document data stream to at least one consumer plug-in, said consumer plug-in having the ability to perform an action in response to the modification made by said active plug-in;
• and
• not forwarding at least some of the document data stream to a subsequent plug-in based on the analysis performed by the active plug-in.

A computer-readable medium having computer-executable instructions for retrieving and processing information from a computer network, wherein retrieving and processing information from a computer network includes performing a Web crawl, wherein performing a Web crawl comprises:

• retrieving an electronic document copy from the computer network in a gathering project, the electronic document copy having text chunks and properties;
• passing the electronic document copy to an active plug-in that analyzes the electronic document copy;
• making at least one change to the electronic document copy with the active plug-in, wherein making the change comprises:
• adding a property to the electronic document copy if the active plug-in determines that a property should be added;
• deleting a property from the electronic document copy if the active plug-in determines that a property should be deleted;
• modifying a property of the electronic document copy if the active plug-in determines that a property should be modified;
• deleting a text chunk from the electronic document copy if the active plug-in determines that the text chunk should be deleted;
• and
• passing the electronic document copy that has been changed by the active plug-in to a consumer plug-in that processes the electronic document copy responsive to the change made in the electronic document copy by the active plug-in.

A system for retrieving and processing information stored on a computer, the system comprising:

• an information retrieval component that retrieves information from the computer;
• at least one modular active plug-in component, each of the modular active plug-in components being associated with the information retrieval component in a sequence that defines the order that the information is passed to each of the at least one modular active plug-in components;
• and
• each of the at least one modular active plug-in components being capable of analyzing and modifying the information before passing the information to a next active plug-in component in the sequence.

A system for retrieving information from a computer network having a plurality of electronic documents stored thereon, wherein each electronic document corresponds to a corresponding document address specification that provides information for locating the electronic document, the system comprising:

• means for retrieving a plurality of electronic documents, each electronic document having content comprising data and meta-tags;
• first computer-executable instructions implemented as an active plug-in object for conducting an analysis of the content of each electronic document and means for modifying the meta-tags associated with each electronic document responsive to the analysis;
• and
• second computer-executable instructions implemented as a consumer plug-in object for processing each electronic document responsive to the meta-tags associated with each electronic document.