Attribute extraction using limited training data
AssigneeYahoo! Inc.
InventorsDmitri Y. Pavlov, Ramnath Balasubramanyan
Life of this patent
Estimating the term…
Abstract
Techniques are described for reducing the false positive rate of regular expression attribute extractions via a specific data representation and a machine learning method that can be trained at a much lower cost (much fewer labeled examples) than would be required by a full scale machine learning solution. Attribute determinations made using the regular expression technique are represented as skeleton tokens. The skeleton tokens, along with accurate attribute determinations, are provided to a machine-learning mechanism to train the machine-learning mechanism. Once trained, the machine-learning mechanism is used to predict the accuracy of attribute determinations represented by skeleton tokens generated for not-yet-analyzed input text.
Claims20
A method, comprising:
- for each sequence of a plurality of attribute determination sequences in a training set, receiving a subset of attribute determinations in the attribute determination sequence that are likely to be a false positive;
- receiving input text, wherein at least a subset of tokens within the input text are a set of attribute tokens with corresponding attribute determinations;
- identifying at least one attribute determination corresponding to an attribute token of the set of attribute tokens as a false positive based on the training set;
- producing a set of filtered attribute tokens by filtering from said set of attribute tokens, all attribute tokens corresponding to attribute determinations that have been identified as false positives,
- wherein the method is performed by a computing device comprising a processor.
A computer-readable medium, comprising volatile or non-volatile memory, carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform steps comprising:
- for each sequence of a plurality of attribute determination sequences in a training set, receiving a subset of attribute determinations in the attribute determination sequence that are likely to be a false positive;
- receiving input text, wherein at least a subset of tokens within the input text are a set of attribute tokens with corresponding attribute determinations;
- identifying at least one attribute determination corresponding to an attribute token of the set of attribute tokens as a false positive based on the training set;
- producing a set of filtered attribute tokens by filtering from said set of attribute tokens, all attribute tokens corresponding to attribute determinations that have been identified as false positives.
Dependencies are inferred from the claim text.