US 10,872,601 B1Granted

Natural language processing

AssigneeAmazon Technologies, Inc.

InventorsAnish Acharya, Angeliki Metallinou, Rahul Goel, Inderjit Dhillon

Life of this patent

Estimating the term…

Loading patent…

Abstract

A natural language understanding (NLU) system that uses a reduced dimensionality of word embedding features to configure compressed NLU models that use reduced computing resources for NLU tasks. A modified NLU model may include a compressed vocabulary data structure of word embedding data vectors that include a set of values corresponding to a reduced dimensionality of the original word embedding features, resulting in a smaller sized vocabulary data structure and reduced size of the vocabulary data structure. Further components of the modified NLU model perform matrix operations to expand the dimensionality of the reduced word embedding data vectors up to the expected dimensionality of later layers of the NLU model. Additional training and reweighting can adjust for potential loses in performance resulting from reductions in the word embedding features. Thus the modified NLU model can achieve similar performance to an original NLU model with reductions in use of computing resources.

Claims21

Dependent claims

A computer-implemented method comprising:

receiving a word string including at least a first word;
selecting a first model to perform a natural language understanding (NLU) operation, the first model including:
a first layer including a data structure corresponding to a plurality of words potentially used with the NLU operation, the data structure including a plurality of data vectors, each data vector corresponding to a respective word and to a first dimensionality of word embedding data,
a second layer including a first matrix configured to be used to transform data having the first dimensionality to data having a second dimensionality, the second dimensionality corresponding to a pre-configured word embedding format, and
a plurality of additional layers including at least:
an initial layer configured to be used to operate on input word embedding data having the second dimensionality, and
a final layer configured to be used to output NLU data;
processing the word string to determine a second matrix;
multiplying the data structure by the second matrix to determine at least a third matrix including first word embedding data corresponding to the first word, the first word embedding data having the first dimensionality;
multiplying the third matrix by the first matrix to determine a fourth matrix, the fourth matrix including second word embedding data corresponding to the first word, the second word embedding data having the second dimensionality;
processing the fourth matrix using the plurality of additional layers to determine first NLU output data;
and
causing an action to be performed using the first NLU output data.

10 more elements

A system comprising:

at least one processor;
and
at least one memory including instructions that, when executed by the at least one processor, cause the system to:
receive text data corresponding to at least a first word;
select a model for performing natural language understanding (NLU) processing, the model including a first portion and a second portion, the second portion including a plurality of layers wherein:
an initial layer of the plurality of layers is configured to be used to operate on input word embedding data having a first dimensionality, and
a final layer of the plurality of layers is configured to be used to output NLU data;
process the text data using the first portion to obtain a first data vector corresponding to the first word, the first data vector including word embedding data having a second dimensionality, the second dimensionality being smaller than the first dimensionality;
process the first data vector to obtain a second data vector corresponding to the first word, the second data vector having the first dimensionality;
process the second data vector using the initial layer of the plurality of layers;
and
determine, as a result of an operation using the final layer of the plurality of layers, first output NLU data corresponding to at least the first word.

9 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/144,197Filed 27 Sep 2018
Granted: 22 Dec 2020
Term ends: Not estimatedThe source records don’t support an estimate.

Classification11 codes

G06F 40/295
Phrasal analysis, e.g. finite state techniques or chunking: named entity recognition
G10L 15/1815
Using natural language modelling: semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
G06F 40/205
Natural language analysis: parsing
G06F 40/30
Handling natural language data: semantic analysis

Family

No related US applications are recorded in the indexed filing history.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key