Back to results
US 12,596,890 B2Granted
Systems and methods for cross-lingual transfer learning
AssigneeSalesforce, Inc.
InventorsLifu Tu, Yingbo Zhou, Caiming Xiong, Jin Qu
Life of this patent
Estimating the term…
Abstract
Embodiments described herein provide a method of training a language model by tuning a prompt. The method comprises masking tokens of first and second conversational texts which have the same semantic meaning but in different languages (e.g., a translation). The masked texts are input to a language model with a prepended soft prompt. The language model generates respective predicted outputs. A loss objective is computed including a masked language model loss. The prompt is updated based on the computed loss objective via backpropagation while keeping the language model frozen.
Claims17
A method of training a neural network based language model to conduct a cross-language conversation by tuning a vector input, the method comprising:
- receiving, via a data interface, a first conversational text in a first natural language, a second conversational text in a second natural language different from the first natural language, and a third conversational text,
- wherein the first conversational text contains a first set of tokens, the second conversational text has a same semantic meaning as the first conversational text and contains a second set of tokens, and the third conversational text is semantically unaligned with the first conversational text and contains a third set of tokens;
- masking a subset of the first set of tokens, a subset of the second set of tokens, and a subset of the third set of tokens, thereby resulting in a masked first set of tokens, a masked second set of tokens, and a masked third set of tokens respectively;
- generating, by the neural network based language model implemented on one or more hardware processors:
- a first predicted output based on a first input of the masked first set of tokens prepended with the vector input,
- a second predicted output, based on a second input of the masked second set of tokens prepended with the vector input, and
- a third predicted output based on a third input of the masked third set of tokens prepended with the vector input;
- computing a loss objective including at least:
- a first masked language model (MLM) loss based on a comparison of the first predicted output and the first conversational text using the first input,
- a second MLM loss based on a comparison of the second predicted output and the second conversational text using the second input, and
- a comparison of representations of the first input and the third input as a negative pair;
- updating the vector input based on the computed loss objective via backpropagation while keeping the neural network based language model frozen;
- and
- implementing the neural network based language model to generate a response using an input combining the updated vector input and a conversational input in the first natural language or the second natural language.
11 more elements
A system for training a neural network based language model to conduct a cross-language conversation by tuning a vector input, the system comprising:
- a memory that stores the neural network based language model and a plurality of processor executable instructions;
- a communication interface that receives a first conversational text in a first natural language, a second conversational text in a second natural language different from the first natural language, and a third conversational text, wherein the first conversational text contains a first set of tokens, the second conversational text has a same semantic meaning as the first conversational text and contains a second set of tokens, and the third conversational text is semantically unaligned with the first conversational text and contains a third set of tokens;
- and
- one or more hardware processors that read and execute the plurality of processor executable instructions from the memory to perform operations comprising:
- masking a subset of the first set of tokens, a subset of the second set of tokens, and a subset of the third set of tokens, thereby resulting in a masked first set of tokens, a masked second set of tokens, and a masked third set of tokens respectively;
- generating, by the neural network based language model implemented on one or more hardware processors:
- a first predicted output based on a first input of the masked first set of tokens prepended with the vector input,
- a second predicted output, based on a second input of the masked second set of tokens prepended with the vector input, and
- a third predicted output based on a third input of the masked third set of tokens prepended with the vector input;
- computing a loss objective including at least:
- a first masked language model (MLM) loss based on a comparison of the first predicted output and the first conversational text using the first input,
- a second MLM loss based on a comparison of the second predicted output and the second conversational text using the second input, and
- a comparison of representations of the first input and the third input as a negative pair;
- updating the vector input based on the computed loss objective via backpropagation while keeping the neural network based language model frozen;
- and
- implementing the neural network based language model to generate a response using an input combining the updated vector input and a conversational input in the first natural language or the second natural language.
13 more elements
A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:
- receiving, via a data interface, a first conversational text in a first natural language, a second conversational text in a second natural language different from the first natural language, and a third conversational text,
- wherein the first conversational text contains a first set of tokens, the second conversational text has a same semantic meaning as the first conversational text and contains a second set of tokens, and the third conversational text is semantically unaligned with the first conversational text and contains a third set of tokens;
- masking a subset of the first set of tokens, a subset of the second set of tokens, and a subset of the third set of tokens, thereby resulting in a masked first set of tokens, a masked second set of tokens, and a masked third set of tokens respectively;
- generating, by a neural network based language model implemented on one or more hardware processors;
- a first predicted output based on a first input of the masked first set of tokens prepended with a vector input,
- a second predicted output, based on a second input of the masked second set of tokens prepended with a vector input, and
- a third predicted output based on a third input of the masked third set of tokens prepended with the vector input;
- computing a loss objective including at least:
- a first masked language model (MLM) loss based on a comparison of the first predicted output and the first conversational text using the first input,
- a second MLM loss based on a comparison of the second predicted output and the second conversational text using the second input, and
- a comparison of representations of the first input and the third input as a negative pair;
- updating the vector input based on the computed loss objective via backpropagation while keeping the neural network based language model frozen;
- and
- implementing the neural network based language model to generate a response using an input combining the updated vector input and a conversational input in the first natural language or the second natural language.
11 more elements
Dependencies are inferred from the claim text.