Contrastive meta-learning for zero-shot learning
AssigneeSap SE
InventorsTassilo Klein, Moin Nabi
Life of this patent
Estimating the term…
Abstract
Disclosed herein are system, method, and computer program product embodiments for utilizing non-RAM memory to implement machine learning configured with a meta-learning training set (small dataset), to create a common-sense predictive language model, thus boosting the performance for downstream tasks. An embodiment operates by receiving a base sentence and perturbation sentences as an input and tokenizing the input to generate a sequence of tokens. Tokens of the semantic perturbation sentences are embedded with tokens of the base sentence as contextually similar tokens pairs to generate training data and classified to capture relationships of the base sentence and the perturbation sentences to generate a classification, which is used to train a language model.
Claims20
A computer implemented method for natural language processing, the method comprising:
- receiving, by a tokenization module, a base sentence and one or more sentences comprising a semantic perturbation of the base sentence as an input, wherein the semantic perturbation of the base sentence comprises one or more linguistic deviations of the base sentence from a first version;
- tokenizing, by the tokenization module, the input to generate a sequence of tokens;
- embedding, by a machine learning engine, tokens of the semantic perturbation with tokens of the base sentence as tokens pairs to generate training data;
- classifying, by a classifier, the semantic perturbation of the token pairs to capture relationships of the base sentence and the one or more sentences to generate a classification;
- and
- training, by the machine learning engine, a language model based at least in part on the training data and the classification;
- and
- wherein at least one of the receiving, tokenizing, determining, embedding and training are performed by one or more computers.
A system, comprising:
- a memory;
- and
- at least one processor coupled to the memory and configured to:
- receive a base sentence and one or more sentences comprising a semantic perturbation of the base sentence as an input, wherein the semantic perturbation of the base sentence comprises one or more linguistic deviations of the base sentence from a first version;
- tokenize the input to generate a sequence of tokens;
- embed tokens of the semantic perturbation with tokens of the base sentence as tokens pairs to generate training data;
- classify the semantic perturbation of the token pairs to capture relationships of the base sentence and the one or more sentences to generate a classification;
- and
- train a language model based, at least in part, on the training data and the classification.
A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:
- receiving a base sentence and one or more sentences comprising a semantic perturbation of the base sentence as an input, wherein the semantic perturbation of the base sentence comprises one or more linguistic deviations of the base sentence from a first version:
- tokenizing the input to generate a sequence of tokens;
- embedding tokens of the semantic perturbation with tokens of the base sentence as tokens pairs to generate training data;
- classifying the semantic perturbation of the token pairs to capture relationships of the base sentence and the one or more sentences to generate a classification;
- and
- training a language model based at least in part on the training data and the classification.
Dependencies are inferred from the claim text.