Back to results
US 8,527,436 B2Granted
Automated parsing of e-mail messages
AssigneeStratify, Inc.
InventorsVamsi Salaka, Joy Thomas
Life of this patent
Estimating the term…
Abstract
An automated parser for e-mail messages identifies component parts such as header, body, signature, and disclaimer. The parser uses a hidden Markov model (HMM) in which the lines making up an e mail are treated as a sequence of observations of a system that evolves according to a Markov chain having states corresponding to the component parts. The HMM is trained using a manually-annotated set of e-mail messages, then applied to parse other e-mail messages. HMM-based parsing can be further refined or expanded using heuristic post-processing techniques that exploit redundancy of some component parts (e.g., signatures, disclaimers) across a corpus of e-mail messages.
Claims23
A method comprising:
- training, in a computer system, an e-mail analyzer that is based on a hidden Markov model, wherein the training uses a set of training e-mail messages, each training e-mail message having a plurality of lines, wherein the hidden Markov model models associations between lines of an e-mail message and a plurality of line types, wherein the plurality of line types include at least a signature type and a disclaimer type;
- applying, in the computer system, the trained e-mail analyzer to a target e-mail message having a plurality of lines to assign each line of the target e-mail message to one of the plurality of line types;
- and
- storing annotations for the target e-mail message in a data store, the annotations indicating the assigned line type for each line of the target e-mail message.
1 more element
A method comprising:
- training, in a computer system, an e-mail analyzer that is based on a hidden Markov model, wherein the training uses a set of training e-mail messages, each training e-mail message having a plurality of lines, wherein the hidden Markov model models associations between lines of an e-mail message and a plurality of line types;
- applying, in the computer system, the trained e-mail analyzer to a target e-mail message having a plurality of lines to assign each line of the target e-mail message to one of the plurality of line types;
- and
- storing annotations for the target e-mail message in a data store, the annotations indicating the assigned line type for each line of the target e-mail message,
- wherein training the e-mail analyzer includes:
- receiving information relating to manual association of each of the lines of each of the training e-mail messages with one of the plurality of line types;
- creating a feature vector representation for each line of each of the training e-mail messages;
- and
- optimizing parameters associated with the hidden Markov model, wherein the optimizing uses a forward backward algorithm applied to the manual associations and the feature vector representations, and
- wherein applying the trained e-mail analyzer includes:
- creating a feature vector representation for each line of the target e-mail message;
- and
- using a Viterbi algorithm and the optimized parameters associated with the hidden Markov model to determine a most probable line type for each line of the target e-mail message, wherein the line is assigned to the most probable line type.
10 more elements
A method comprising:
- training, in a computer system, an e-mail analyzer that is based on a hidden Markov model, wherein the training uses a set of training e-mail messages, each training e-mail message having a plurality of lines, wherein the hidden Markov model models associations between lines of an e-mail message and a plurality of line types;
- applying, in the computer system, the trained e-mail analyzer to a target e-mail message having a plurality of lines to assign each line of the target e-mail message to one of the plurality of line types;
- storing annotations for the target e-mail message in a data store, the annotations indicating the assigned line type for each line of the target e-mail message;
- and
- establishing a definition for a feature vector usable to represent a line of an e-mail message, wherein the feature vector includes a plurality of bits, each bit indicating presence or absence of an associated one of a plurality of features,
- wherein the training of the e-mail analyzer includes generating a feature vector for each line of the training e-mail messages using the feature vector definition and the applying of the trained e-mail analyzer includes generating a feature vector for each line of the target e-mail message using the feature vector definition.
3 more elements
A method comprising:
- training, in a computer system, an e-mail analyzer that is based on a hidden Markov model, wherein the training uses a set of training e-mail messages, each training e-mail message having a plurality of lines, wherein the hidden Markov model models associations between lines of an e-mail message and a plurality of line types;
- applying, in the computer system, the trained e-mail analyzer to a plurality of target e-mail messages, each target e-mail message having a plurality of lines, to assign each line of each of the target e-mail messages to one of the plurality of line types;
- extracting, from each of the plurality of target e-mail messages, a block of consecutive lines that were each assigned to a first one of the line types;
- forming a plurality of clusters of the blocks based on similarity of lines in different blocks;
- and
- for each of the plurality of clusters, determining a representative block.
3 more elements
A computer system comprising:
- a document information data store configured to store information about a plurality of e-mail messages;
- a processor coupled to the document information data store, the processor being configured to:
- train an e-mail analyzer that is based on a hidden Markov model, wherein the training uses a set of training e-mail messages, each training e-mail message having a plurality of lines, wherein the hidden Markov model models associations between lines of an e-mail message and a plurality of line types;
- apply the trained e-mail analyzer to a plurality of target e-mail messages, each target e-mail message having a plurality of lines, to assign each line of each of the target e-mail messages to one of the plurality of line types;
- extract, from each of the plurality of target e-mail messages, a block of consecutive lines that were each assigned to a first one of the line types;
- form a plurality of clusters of the blocks based on similarity of lines in different blocks;
- and
- for each of the plurality of clusters, determine a representative block.
5 more elements
A method comprising:
- analyzing, in a computer system, a plurality of e-mails from a sender using an e-mail analyzer based on a hidden Markov model, wherein the analyzing includes associating, based on the hidden Markov model, each line of an e-mail with one of a plurality of line types, the plurality of line types including a signature type;
- extracting, in the computer system, from the plurality of e-mails a plurality of signature blocks, wherein each signature block includes a plurality of contiguous lines of the e-mail that are each associated with the signature type;
- validating, in the computer system, the signature blocks;
- forming, in the computer system, one or more clusters of signature blocks based on a degree of similarity thereof;
- and
- determining, in the computer system, a most likely signature for the sender based on the one or more clusters.
3 more elements
Dependencies are inferred from the claim text.