Back to results
US 12,748,932 B2Granted
Compression of word embeddings for natural language processing systems
AssigneeMicrosoft Technology Licensing, LLC
InventorsXihui Lin, Andrew James Mcnamara, Kaheer Suleman
Life of this patent
Estimating the term…
Abstract
Described herein are systems and methods that provide a natural language processing system (NLPS) that employs compressed word embeddings. An auto-encoder that includes encoder circuitry and decoder circuitry can be used to produce the compressed word embeddings. The decoder circuitry is trained to decompress the word embeddings with reduced or minimal differences between the original uncompressed word embeddings and the corresponding decompressed word embeddings. One or more parameters of the trained decoder circuitry are transferred to the NLPS, where the NLPS is then trained using the compressed word embeddings to improve the correctness of the responses or actions determined by the NLPS.
Claims19
A computing system, comprising:
- at least one processor;
- and
- a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations comprising:
- receiving a spoken language input via a speech-to-text application;
- obtaining original uncompressed word embeddings for the spoken language input, the original uncompressed word embeddings including a vector of real numbers;
- generating input text compressed word embeddings based on the uncompressed word embeddings using an encoder of a trained auto-encoder, the input text compressed word embeddings including a vector of binary numbers;
- processing the input text compressed word embeddings with a neural network to generate output compressed word embeddings including a vector of binary numbers;
- decompressing the output compressed word embeddings to produce one or more decompressed word embeddings using a decoder of the trained auto-encoder;
- and
- outputting a natural language output based on the one or more decompressed word embeddings, using a text-to-speech application.
7 more elements
A method, comprising:
- obtaining a first set of one or more uncompressed word embeddings;
- obtaining a first set of one or more compressed word embeddings corresponding to the first set of one or more uncompressed word embeddings from an encoder of an auto-encoder;
- adjusting a set of parameters to reduce differences between a first set of decompressed word embeddings corresponding to the first set of one or more compressed word embeddings and the first set of one or more uncompressed word embeddings;
- training a decoder of the auto-encoder separately from the encoder using the set of parameters;
- and
- storing, for subsequent processing by the natural language processor, the first set of one or compressed word embeddings and the set of parameters, wherein the method further comprises, during training:
- training the natural language processor using a second set of one or more uncompressed word embeddings and generating one or more parameters of the natural language processor;
- obtaining a second set of one or more compressed word embeddings that correspond to the second set of one or more uncompressed word embeddings and the set of parameters of the decoder of the auto-encoder;
- updating at least one of one or more parameters of the natural language processor with at least one of the set of parameters of the decoder of the auto-encoder;
- and
- retraining the natural language processor using the second set of one or more compressed word embeddings and the updated one or more parameters of the natural language processor.
8 more elements
A method, comprising:
- receiving, by a natural language processor, an input;
- obtaining, by the natural language processor and from a storage device, one or more compressed word embeddings based on the input, wherein the one or more compressed word embeddings were generated by an encoder of an auto-encoder prior to receiving the input;
- decompressing, by the natural language processor, the one or more compressed word embeddings to produce one or more decompressed word embeddings, wherein the storage device further comprises one or more parameters that are generated by a decoder of the auto-encoder, and wherein the natural language processor and the decoder were trained separately from the encoder;
- and
- determining, by the natural language processor, an action to be performed in response to the input based on the one or more decompressed word embeddings, wherein the method further comprises, during training:
- training the natural language processor using one or more uncompressed word embeddings and generating one or more parameters of the natural language processor;
- obtaining one or more compressed word embeddings that correspond to the one or more uncompressed word embeddings and a set of parameters of the decoder of the auto-encoder;
- updating at least one of one or more parameters of the natural language processor with at least one of the set of parameters of the decoder of the auto-encoder;
- and
- retraining the natural language processor using the one or more compressed word embeddings and the updated one or more parameters of the natural language processor.
7 more elements
Dependencies are inferred from the claim text.