System for automatically managing duplicate documents when crawling dynamic documents

A computer readable storage medium storing one or more programs for execution by one or more processors, the one or more programs comprising:

• a partitioning module for partitioning a plurality of document identifiers into multiple clusters, the document identifiers comprising universal resource locators (URLs), wherein each document identifier is a text string that identifies exactly one document, each cluster has a cluster name, and each cluster has a set of URL parameters, wherein each URL parameter is a text string contained within the URLs in the cluster;
• an equivalence rule generator for generating an equivalence rule for a cluster of URLs, the equivalence rule specifying which of the URL parameters associated with the cluster are content-relevant;
• and
• a grouping module for grouping a respective cluster into a plurality of equivalence classes in accordance with its equivalence rule, each equivalence class including URLs that correspond to a document content associated with the equivalence class wherein all the URLs in a respective equivalence class of the plurality of equivalence classes have the same hostname and reference documents having substantially the same content;
• an identifying module for identifying a single one of the URLs having the same hostname within each equivalence class as its representative document identifier;
• and
• an operation module for performing, with respect to a respective equivalence class having a plurality of URLs, a particular computer-implemented operation on only the single representative URL of the equivalence class.