System and method for efficiently processing messages stored in multiple message stores

A method for efficiently identifying unique messages stored in organized message stores, comprising:

• removing duplicate messages containing substantially duplicative content from topically identical messages logically extracted from a plurality of organized message stores;
• removing near-duplicate messages containing content recursively included within another of the remaining messages;
• and
• storing unique messages comprising at least one of a message store a single occurrence of a given topic and a message storing non-recursive content relative to each other such logically extracted message.

A system for efficiently processing messages stored in multiple message stores, comprising:

• a message processor iteratively copying metadata identifying a range of topically identical messages extracted from a plurality of message stores storing a multiplicity of messages to be processed and categorizing the metadata for the extracted range of topically identical messages, the message process further comprising:
• a duplicate message selector identifying those messages containing substantially duplicative content within the extracted range as duplicate messages;
• a thread length selector tallying those non-duplicate messages within the extracted range into an ordering of conversation thread length;
• a near-duplicate message selector classifying those messages whose content is recursively-included content within another of the tallied non-duplicate messages as near-duplicate messages;
• and
• a unique message selector designating the remaining messages as unique messages containing substantially non-duplicative content.

A method for efficiently processing messages stored in multiple message stores, comprising:

• iteratively copying metadata identifying a range of topically identical messages extracted from a plurality of message stores storing a multiplicity of messages to be processed;
• and
• categorizing the metadata for the extracted range of topically identical messages, comprising:
• identifying those messages containing substantially duplicative content within the extracted range as duplicate messages;
• tallying those non-duplicate messages within the extracted range into an ordering of conversation thread length;
• classifying those messages whose content is recursively-included content within another of the tallied non-duplicate messages as near-duplicate messages;
• and
• designating the remaining messages as unique messages containing substantially non-duplicative content.

A system for categorizing messages stored in message stores into discrete categories, comprising:

• a master array storing metadata for each message to be processed from a plurality of message stores, the metadata identifying the source message store and relative storage location for the message;
• means for sorting the metadata according to topic and comparing content of messages with similar topics to identify those messages containing substantially duplicative content;
• means for sorting the messages according to content by referencing the metadata and ordering the metadata in order of conversation thread length;
• means for comparing the content to identify those messages whose content is recursively-included content within another of the messages;
• and
• means for identifying the remaining messages by referencing the metadata as unique messages.

A method for categorizing messages stored in message stores into discrete categories, comprising:

• extracting metadata for each message to be processed from a plurality of message stores, the metadata identifying the source message store and relative storage location for the message;
• sorting the metadata according to topic and comparing content of messages with similar topics to identify those messages containing substantially duplicative content;
• sorting the messages according to content by referencing the metadata and ordering the metadata in order of conversation thread length;
• comparing the content to identify those messages whose content is recursively-included content within another of the messages;
• and
• identifying the remaining messages by referencing the metadata as unique messages.