Pre-Trained Projection Networks for Transferable Natural Language Representations
AssigneeGoogle Inc.
InventorsSujith Ravi, Zornitsa Kozareva, Chinnadhurai Sankar
Life of this patent
Estimating the term…
Abstract
Systems and methods are provided to pre-train projection networks for use as transferable natural language representation generators. In particular, example pre-training schemes described herein enable learning of transferable deep neural projection representations over randomized locality sensitive hashing (LSH) projections, thereby surmounting the need to store any embedding matrices because the projections can be dynamically computed at inference time.
Claims20
A computing system, comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media that collectively store:
- a pre-trained projection network configured to receive a language input comprising one or more units of text and to dynamically generate an intermediate representation from the language input, the projection network comprising:
- a sequence of one or more projection layers, wherein each projection layer is configured to receive a layer input and apply a plurality of projection layer functions to the layer input to generate a projection layer output;
- and
- a sequence of one or more intermediate layers configured to receive the projection layer output generated by a last projection layer in the sequence of one or more projection layers and to generate one or more intermediate layer outputs, wherein the intermediate representation comprises the intermediate layer output generated by a last intermediate layer in the sequence of one or more intermediate layers;
- instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
- obtaining the language input;
- inputting the language input into the pre-trained projection network;
- and
- receiving the intermediate representation as an output of the pre-trained projection network.
A computer-implemented method to pre-train a projection network comprising one or more projection layers and one or more intermediate layers, each projection layer configured to apply one or more projection functions to project a layer input into a different dimensional space, the projection network configured to receive an input and to generate an intermediate representation for the input, the method comprising:
- accessing, by one or more computing devices, a set of training data comprising a plurality of example inputs;
- inputting, by the one or more computing devices, each of the plurality of example inputs into the projection network;
- receiving, by the one or more computing devices, a respective intermediate representation for each of the plurality of example inputs as an output of the projection network;
- inputting, by the one or more computing devices, each respective intermediate representation into a decoder model configured to reconstruct inputs based on intermediate representations;
- receiving, by the one or more computing devices, a respective reconstructed input for each of the plurality of example inputs as an output of the decoder model;
- and
- learning, by the one or more computing devices, one or more parameter values for the one or more intermediate layers of the projection network based at least in part on a comparison of each respective reconstructed input to the corresponding example input.
A computer-implemented method to pre-train a projection network comprising one or more projection layers and one or more intermediate layers, each projection layer configured to apply one or more projection functions to project a layer input into a different dimensional space, the projection network configured to receive an input and to generate an intermediate representation for the input, the method comprising:
- accessing, by one or more computing devices, a set of training data comprising a plurality of input words, wherein a respective set of ground truth context words are associated with each of the plurality of input words;
- inputting, by the one or more computing devices, each of the plurality of input words into the projection network;
- receiving, by the one or more computing devices, a respective intermediate representation for each of the plurality of input words as an output of the projection network;
- determining, by the one or more computing devices, a set of predicted context words for each of the plurality of input words based at least in part on the respective intermediate representation for each of the plurality of input words;
- and
- learning, by the one or more computing devices, one or more parameter values for the one or more intermediate layers of the projection network based at least in part on a comparison, for each input word, of the respective set of predicted context words to the respective set of ground truth context words.
Dependencies are inferred from the claim text.