Unsupervised method to generate annotations for natural language understanding tasks
AssigneeMicrosoft Technology Licensing, LLC
InventorsHany Mohamed Hassan AWADALLA, Subhabrata Mukherjee, Ahmed Awadallah
Life of this patent
Estimating the term…
Abstract
A method for training a machine learning model with parallel annotations of source instances and while facilitating security of the source instances can be performed by a system that generates a coupled machine learning model from (1) a first machine learning model trained on a first set of training data comprising unannotated natural language and (2) a second machine learning model trained on populated target templates which are populated with a plurality of vocabulary words. Once formed, the coupled machine learning model is configured to transform unannotated natural language into annotated machine-readable text.
Claims20
A computer implemented method for training a machine learning model with generated annotations of source instances and while facilitating security of the source instances, the method being implemented by a computing system that includes at least one hardware processor and the method comprising:
- the computing system receiving electronic content comprising (i) a set of target templates comprising a plurality of keys and (ii) a set of vocabulary words comprising a plurality of values, the plurality of values corresponding to the plurality of keys;
- the computing system automatically populating the set of target templates with the set of vocabulary words to generate training data comprising synthetically populated target templates of key-value pairings formatted as annotated machine-readable text;
- the computing system training a machine learning model with the training data, the machine learning model being configured to understand an association between the plurality of keys and the plurality of values of the key-value pairings included in the populated target templates;
- and
- the computing system combining the machine learning model with a different machine learning model that was trained to understand a semantic structure of unannotated natural language, the machine learning model and the different machine learning model being combined into a coupled machine learning model by aligning word embeddings output from the machine learning model and word embeddings output from the different machine learning model, the coupled machine learning model being configured to transform new unannotated natural language into annotated machine-readable text.
A computing system comprising:
- one or more processors;
- one or more computer-readable hardware storage devices having stored thereon computer-executable instructions that are executable by the one or more processors to cause the computer system to at least:
- receive electronic content comprising (i) a set of target templates comprising a plurality of keys and (ii) a set of vocabulary words comprising a plurality of values, the plurality of values corresponding to the plurality of keys;
- automatically populate the set of target templates with the set of vocabulary words to generate training data comprising synthetically populated target templates of key-value pairings formatted as annotated machine-readable text;
- train a machine learning model with the training data, the machine learning model being configured to understand an association between the plurality of keys and the plurality of values of the key-value pairings included in the populated target templates;
- combine the machine learning model and the different machine learning model into a coupled machine learning model, the coupled machine learning model being configured to transform unannotated natural language into annotated machine-readable text;
- and
- operating the coupled machine learning model to generate annotated machine-readable text directly from unannotated natural language and without requiring a use of any intermediate annotated representation when generating the annotated machine-readable text directly from the unannotated natural language.
A computer implemented method for training a machine learning model with generated annotations of source instances and while facilitating security of the source instances, the method being implemented by a computing system that includes at least one hardware processor and the method comprising:
- the computing system receiving a set of training data comprising a plurality of source instances corresponding to a particular language;
- the computing system training a machine learning model with the training data, the machine learning model being configured to understand a semantic structure of the training data in the particular language;
- the computing system combining the machine learning model with a different machine learning model into a coupled machine learning model by aligning word embeddings output from the first machine learning model and word embeddings output from the different machine learning model, the coupled machine learning model being configured to transform source instances in a first language that the different machine learning model is trained to understand the semantic structure of into source instances in the particular language that the machine learning model is trained to understand the semantic structure of.
One or more computer-readable hardware storage devices having stored thereon computer-executable instructions that are executable by the one or more processors to cause a computer system to at least:
- operate a coupled machine learning model configured to transform unannotated natural language into machine-readable text;
- and
- transform unannotated natural language into machine readable text;
- wherein the coupled machine learning model comprises:
- a first machine learning model trained on a first set of training data comprising unannotated natural language, the first machine learning model configured to understand a semantic structure of the first set of data;
- and
- a second machine learning model trained on a second set of training data comprising a plurality of target templates populated with a plurality of values, the plurality of target templates comprising a plurality of keys corresponding to the plurality of values;
- wherein the second machine learning model is configured to understand an association between the plurality of keys and the plurality of values of one or more key-value pairings included in the populated target templates;
- and
- wherein the first machine learning model and second machine learning model are combined to form the coupled machine learning model which further comprises a source decoder trained to decode unannotated natural language and a target decoder trained to decode target templates, and an encoder shared between the source decoder and the target decode.
Dependencies are inferred from the claim text.