Back to results
US 8,527,500 B2Granted
Preprocessing text to enhance statistical features
AssigneeRed Hat, Inc.
InventorJames Paul Schneider
Life of this patent
Estimating the term…
Abstract
A document preprocessor preprocess a document to enhance the statistical features of the document. The system preprocesses the document by matching a prefix and a trailing context in the document with one or more matching prefixes in a transformation database, where the prefix is a first string of one or more tokens in the first document and the trailing context is a second string of one or more tokens in the first document that trail the prefix. Alternatively, the system preprocesses the document by computing cyclic permutations of the document, sorting these permutations and taking the last token from each of the sorted permutations.
Claims18
A computer-implemented method comprising:
- retrieving, by a processing device, a first document that is subsequently compared with a second document;
- preprocessing, with a document preprocessor executable by the processing device, the first document to enhance statistical features of the first document, wherein the preprocessing comprises:
- replacing a prefix in the first document with a semantically similar word of a same language as a language of the prefix, wherein the semantically similar word comprises a different word than the prefix and comprises a meaning of the prefix;
- computing a plurality of cyclic permutations from a plurality of tokens in the first document;
- sorting the plurality of cyclic permutations;
- taking the last token from each permutation of the set of cyclic permutations, and
- outputting a string constructed from the plurality of last tokens;
- transforming the first document into a first preprocessed document by matching the prefix and a trailing context in the first document with matching prefixes in a transformation database, wherein the prefix comprises a first string of tokens of the plurality of tokens in the first document and the trailing context comprises a second string of tokens of the plurality of tokens that trail the prefix in the first document;
- and
- transforming the first preprocessed document into a second preprocessed document by rearranging an order of the plurality of tokens in the first preprocessed document.
7 more elements
A non-transitory computer readable storage medium having instructions stored thereon that, when executed by a processing device, cause the processing device to perform operations comprising:
- retrieving, by the processing device, a first document that is subsequently compared with a second document;
- preprocessing, with a document preprocessor executable by the processing device, the first document to enhance statistical features of the first document, wherein the preprocessing comprises:
- replacing a prefix in the first document with a semantically similar word of a same language as a language of the prefix, wherein the semantically similar word comprises a different word than the prefix and comprises a meaning of the prefix;
- computing a plurality of cyclic permutations from a plurality of tokens in the first document;
- sorting the plurality of cyclic permutations;
- taking the last token from each permutation of the set of cyclic permutations, and
- outputting a string constructed from the plurality of last tokens;
- transforming the first document into a first preprocessed document by matching the prefix and a trailing context in the first document with matching prefixes in a transformation database, wherein the prefix comprises a first string of tokens of the plurality of tokens in the first document and the trailing context comprises a second string of tokens of the plurality of tokens that trail the prefix in the first document;
- and
- transforming the first preprocessed document into a second preprocessed document by rearranging an order of the plurality of tokens in the first preprocessed document.
7 more elements
A document processing system comprising:
- a memory;
- a processing device communicably coupled to the memory;
- a document database communicably coupled to the memory and the processing device, the document database to store a first document that is subsequently compared with a second document;
- and
- a document preprocessor executable from the memory by the processing device, the document preprocessor to:
- preprocess the first document to enhance statistical features of the first document, wherein document preprocessor further comprises an order preprocessor to, as part of the preprocessing:
- compute a plurality of cyclic permutations from a plurality of tokens in the first document;
- sort the plurality of cyclic permutations;
- take the last token from each permutation of the set of cyclic permutations, and
- output a string constructed from the plurality of last tokens;
- replace a prefix in the first document with a semantically similar word of a same language as a language of the prefix, wherein the semantically similar word comprises a different word than the prefix and comprises a meaning of the prefix;
- transform the first document into a first preprocessed document by matching the prefix and a trailing context in the first document with matching prefixes in a transformation database, wherein the prefix comprises a first string of tokens of the plurality of tokens in the first document and the trailing context comprises a second string of tokens of the plurality of tokens that trail the prefix in the first document;
- and
- transform the first preprocessed document into a second preprocessed document by rearranging an order of the plurality of tokens in the first preprocessed document.
11 more elements
Dependencies are inferred from the claim text.