Iterative word list expansion
AssigneeABBYY InfoPoisk LLC
InventorsDaria Bogdanova, Nikolay Kopylov
Life of this patent
Estimating the term…
Abstract
Methods and systems are provided for expanding an electronic word list, containing a set of words where each word is associated with a label from a first set of labels. A subset of training data containing a set of texts having a second set of labels is obtained. For each word in the electronic word list and a label in the sub-set of the training data, a feature selection criterion is calculated. One or more words are selected, for which resulting value of the feature selection criterion calculation is greater than a predetermined threshold value. The one or more selected words are added to the electronic word list.
Claims23
A method for expanding an electronic word list, the method comprising:
- obtaining the electronic word list containing a set of words, wherein each word is associated with a label from a first set of labels;
- obtaining a subset of training data containing a set of texts having a second set of labels;
- for each word in the electronic word list and a label in the sub-set of the training data, calculating a feature selection criterion;
- selecting one or more words for which resulting value of the feature selection criterion calculation is greater than a predetermined threshold value;
- and
- adding the one or more selected words to the electronic word list.
A system comprising:
- one or more data processors;
- and
- one or more storage devices storing instructions that, when executed by the one or more data processors, cause the one or more data processors to perform operations comprising:
- obtaining an electronic word list containing a set of words, wherein each word is associated with a label from a first set of labels;
- obtaining a subset of training data containing a set of texts having a second set of labels;
- for each word in the electronic word list and a label in the sub-set of the training data, calculating a feature selection criterion;
- selecting one or more words for which resulting value of the feature selection criterion calculation is greater than a predetermined threshold value;
- and
- adding the one or more selected words to the electronic word list.
A computer-readable storage medium having machine instructions stored therein, the instructions being executable by a processor to cause the processor to perform operations comprising
- obtaining an electronic word list containing a set of words, wherein each word is associated a label from a first set of labels;
- obtaining a subset of training data containing a set of texts having a second set of labels;
- for each word in the electronic word list and a label in the sub-set of the training data, calculating a feature selection criterion;
- selecting one or more words for which resulting value of the feature selection criterion calculation is greater than a predetermined threshold value;
- and
- adding the one or more selected words to the electronic word list.
Dependencies are inferred from the claim text.