US 7,003,725 B2 · Method and system for normalizing dirty text in a document

Abstract

A method and system of normalizing dirty text in a document. The present invention creates a thesaurus that evolves over time as new document collections are analyzed. This thesaurus, which is used by an editor, contains standard terms and phrases, and their corresponding variations of these standard terms and phrases. Documents are run through this editor and misspelled words or phrases, joined words, and ad hoc abbreviations are replaced with standard terms from the thesaurus. The present invention also enables normalization of documents in cases where a list of standard terms must be inferred from the corpus of the document. The normalizer will facilitate data mining applications which can not function properly with dirty text, resulting in more accurate analysis of documents. Over time, as the thesaurus evolves, collecting more words and phrases, the process of generating the thesaurus will become more automated.

Claims24

Dependent claims

A method for normalizing text in a document, said method comprising:

a) generating a list of reference words and phrases and a list of non-reference words and phrases from a selected group of documents;
b) comparing said list of reference words and phrases with a joined list containing said reference words and phrases and said non-reference words and phrases, using an edit-distance algorithm to create an approximate duplicates list;
c) filtering said approximate duplicates list to create a thesaurus of standard words and phrases and their variations;
and
d) editing said selected group of documents with an editor operable to use said thesaurus to replace a word or phrase on said approximate duplicates list with said standard words and phrases.

2 more elements

A computer system comprising:

a bus;
a memory unit coupled to said bus;
and
a processor coupled to said bus, said processor for executing a method for normalizing text in a document, said method comprising:
a) generating a list of reference words and phrases and a list of non-reference words and phrases from a selected group of documents;
b) comparing said list of reference words and phrases with a joined list containing said reference words and phrases and said non-reference words and phrases using an edit-distance algorithm to create an approximate duplicates list;
c) filtering said approximate duplicates list to create a thesaurus of standard words and phrases and their variations;
and
d) editing said selected group of documents with an editor operable to use said thesaurus to replace a word or phrase on said approximate duplicates list with said standard words and phrases.

6 more elements

A computer-usable medium having computer-readable program code embodied therein for causing a computer system to perform:

a) generating a list of reference words and phrases and a list of non-reference words and phrases from a selected group of documents;
b) comparing said list of reference words and phrases with a joined list containing said reference words and phrases and said non-reference words and phrases using an edit-distance algorithm to create an approximate duplicates list;
c) filtering said approximate duplicates list to create a thesaurus of standard words and phrases and their variations;
and
d) editing said selected group of documents with an editor operable to use said thesaurus to replace a word or phrase on-said approximate duplicates list with said standard words and phrases.

2 more elements

Dependencies are inferred from the claim text.

Method and system for normalizing dirty text in a document

Life of this patent

Method and system for normalizing dirty text in a document

Life of this patent

Abstract

Claims24