US 2022/0067304 A1Published application

Energy-Based Language Models

AssigneeGoogle Inc.

InventorsThang Minh Luong, Quoc V. Le, Kevin Stefan Clark

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems and methods are provided for training and using energy-based language models such as cloze language models. In particular, one aspect of the present disclosure is directed to an energy-based cloze language model for representation learning over text. In some instances, the models provided herein can be referred to as the “Electric” model. Similar to the BERT model, example models proposed herein can be a conditional generative model of tokens given their contexts. However, example models proposed herein do not mask text or output a full distribution over tokens that could occur in a context. Instead, the example proposed models assign a scalar energy score to each input token. Another aspect of the present disclosure provides techniques to train the proposed models to assign low energies to data tokens and high energies to other ones using an algorithm based on noise-contrastive estimation.

Claims20

Dependent claims

A computer-implemented method to train a machine-learned language model, the method comprising:

for each of one or more training iterations:
obtaining, by a computing system comprising one or more computing devices, an original language input that comprises a plurality of positive tokens;
generating, by the computing system, one or more noise tokens;
respectively replacing, by the computing system, one or more of the plurality of positive tokens in the original language input with the one or more noise tokens to form a noised language input that comprises a plurality of updated input tokens;
processing, by the computing system, the noised language input with the machine-learned language model to produce a plurality of scores respectively for the plurality of updated input tokens, wherein the score for each updated input token indicates a likelihood of the updated input token given the other updated input tokens in the noised language input;
generating, by the computing system, a plurality of predictions respectively for the plurality of updated input tokens based at least in part on the plurality of scores, wherein the prediction produced by the machine-learned language model for each updated input token predicts whether such updated input token is a positive token or a noise token;
and
training, by the computing system, the machine-learned language model based at least in part on a loss function that evaluates the plurality of predictions.

5 more elements

One or more non-transitory computer-readable media that store a machine-learned language model produced through performance of operations, the operations comprising:

for each of one or more training iterations:
obtaining, by a computing system, an original language input that comprises a plurality of positive tokens;
generating, by the computing system, one or more noise tokens;
respectively replacing, by the computing system, one or more of the plurality of positive tokens in the original language input with the one or more noise tokens to form a noised language input that comprises a plurality of updated input tokens;
processing, by the computing system, the noised language input with the machine-learned language model to produce a plurality of scores respectively for the plurality of updated input tokens, wherein the score for each updated input token indicates a likelihood of the updated input token given the other updated input tokens in the noised language input;
generating, by the computing system, a plurality of predictions respectively for the plurality of updated input tokens based at least in part on the plurality of scores, wherein the prediction produced by the machine-learned language model for each updated input token predicts whether such updated input token is a positive token or a noise token;
and
training, by the computing system, the machine-learned language model based at least in part on a loss function that evaluates the plurality of predictions.

5 more elements

Dependencies are inferred from the claim text.