Contrastive self-supervised machine learning for commonsense reasoning
AssigneeSap SE
InventorsTassilo Klein, Moin Nabi
Life of this patent
Estimating the term…
Abstract
In an example embodiment, a self-supervised learning task is used for training commonsense-aware representations in a minimally supervised fashion and a pair level mutual-exclusive loss is used to enforce commonsense knowledge during representation learning. This helps to exploit the mutual-exclusive nature of the training samples of commonsense reasoning corpora. Given two pieces of input where the only difference between them are trigger pieces of data, it may be postulated that the pairwise pronoun disambiguation is mutually exclusive. This idea is formulated using a contrastive loss and then this is used to update the language model.
Claims17
A system comprising:
- at least one hardware processor;
- and
- a non-transitory computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
- obtaining training data pairs, each training data pair being a pair of groupings of data having mutual exclusive relationships and identical to each other except for trigger pieces of data, each grouping of data having an ambiguous piece of data;
- feeding each of the training data pairs into a machine learning algorithm using a mutual exclusive loss function to train a commonsense machine learned model, the mutual exclusive loss function being a combination of a first and a second term, the first term being a measurement of mutual exclusivity of potential meanings of ambiguous pieces of data across pairs, the second term being a measurement of mutual exclusivity of potential meanings of ambiguous pieces of data within each pair;
- and
- feeding a grouping of text sentences into the commonsense machine learned model to output a prediction of a resolution of an ambiguous piece of data in the grouping.
A method comprising:
- obtaining training data pairs, each training data pair being a pair of groupings of data having mutual exclusive relationships and identical to each other except for trigger pieces of data, each grouping of data having an ambiguous piece of data;
- feeding each of the training data pairs into a machine learning algorithm using a mutual exclusive loss function to train a commonsense machine learned model, the mutual exclusive loss function being a combination of a first and a second term, the first term being a measurement of mutual exclusivity of potential meanings of ambiguous pieces of data across pairs, the second term being a measurement of mutual exclusivity of potential meanings of ambiguous pieces of data within each pair;
- and
- feeding a grouping of text sentences into the commonsense machine learned model to output a prediction of a resolution of an ambiguous piece of data in the grouping.
A non-transitory machine-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- obtaining training data pairs, each training data pair being a pair of groupings of data having mutual exclusive relationships and identical to each other except for trigger pieces of data, each grouping of data having an ambiguous piece of data;
- feeding each of the training data pairs into a machine learning algorithm using a mutual exclusive loss function to train a commonsense machine learned model, the mutual exclusive loss function being a combination of a first and a second term, the first term being a measurement of mutual exclusivity of potential meanings of ambiguous pieces of data across pairs, the second term being a measurement of mutual exclusivity of potential meanings of ambiguous pieces of data within each pair;
- and
- feeding a grouping of text sentences into the commonsense machine learned model to output a prediction of a resolution of an ambiguous piece of data in the grouping.
Dependencies are inferred from the claim text.