US 10,241,997 B1 · Computing numeric representations of words in a high-dimensional space

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for computing numeric representations of words. One of the methods includes obtaining a set of training data, wherein the set of training data comprises sequences of words; training a classifier and an embedding function on the set of training data, wherein training the embedding function comprises obtained trained values of the embedding function parameters; processing each word in the vocabulary using the embedding function in accordance with the trained values of the embedding function parameters to generate a respective numerical representation of each word in the vocabulary in the high-dimensional space; and associating each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space.

Drawings

Drawings are temporarily unavailable. Open the PDF at USPTO

Drawings

Claims18

Dependent claims

A method performed by one or more computers, the method comprising:

receiving a request for a numeric representation for a particular word in a high-dimensional space;
determining, by accessing a data set that is stored on one or more non-transitory computer readable media and that associates each word in a vocabulary of words with a respective numeric representation of the word in the high-dimensional space, the numeric representation of the particular word;
and
providing the numeric representation of the particular word in response to the request, wherein the data set has been generated by:
obtaining a set of training data, wherein the set of training data comprises sequences of words;
training a plurality of classifiers and an embedding function on the set of training data, wherein the embedding function receives an input word and maps the input word to a numeric representation in the high-dimensional space in accordance with a set of embedding function parameters, wherein each of the classifiers corresponds to a respective position surrounding the input word in a sequence of words, and wherein each of the classifiers processes the numeric representation of the input word to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word will be found in the corresponding position relative to the input word, and wherein training the embedding function comprises determining trained values of the embedding function parameters;
processing each word in the vocabulary using the embedding function in accordance with the trained values of the embedding function parameters to generate a respective numeric representation of each word in the vocabulary;
associating each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space;
and
storing the data set on the one or more non-transitory computer storage media.

7 more elements

A method performed by one or more computers, the method comprising:

receiving a request for a numeric representation for a particular word in a high-dimensional space;
determining, by accessing a data set that is stored on one or more non-transitory computer readable media and that associates each word in a vocabulary of words with a respective numeric representation of the word in the high-dimensional space, the numeric representation of the particular word;
and
providing the numeric representation of the particular word in response to the request, wherein the data set has been generated by:
obtaining a set of training data, wherein the set of training data comprises sequences of words;
training a classifier and an embedding function on the set of training data, wherein the embedding function receives a plurality of words surrounding an unknown word in a sequence of words and maps the plurality of words into a numeric representation in accordance with a set of embedding function parameters, wherein the classifier processes the numeric representation of the sequence of words to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores measure a predicted likelihood that the corresponding word is the unknown word, and wherein training the embedding function comprises determining trained values of the embedding function parameters;
processing each word in the vocabulary using the embedding function in accordance with the trained values of the embedding function parameters to generate a respective numeric representation of each word in the vocabulary in the high-dimensional space;
associating each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space;
and
storing the data set on the one or more non-transitory computer storage media.

7 more elements

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

receiving a request for a numeric representation for a particular word in a high-dimensional space;
determining, by accessing a data set that is stored on one or more non-transitory computer readable media and that associates each word in a vocabulary of words with a respective numeric representation of the word in the high-dimensional space, the numeric representation of the particular word;
and
providing the numeric representation of the particular word in response to the request, wherein the data set has been generated by:
obtaining a set of training data, wherein the set of training data comprises sequences of words;
training a plurality of classifiers and an embedding function on the set of training data, wherein the embedding function receives an input word and maps the input word to a numeric representation in the high-dimensional space in accordance with a set of embedding function parameters, wherein each of the classifiers corresponds to a respective position surrounding the input word in a sequence of words, and wherein each of the classifiers processes the numeric representation of the input word to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word will be found in the corresponding position relative to the input word, and wherein training the embedding function comprises determining trained values of the embedding function parameters;
processing each word in the vocabulary using the embedding function in accordance with the trained values of the embedding function parameters to generate a respective numeric representation of each word in the vocabulary;
associating each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space;
and
storing the data set on the one or more non-transitory computer storage media.

7 more elements

receiving a request for a numeric representation for a particular word in a high-dimensional space;
determining, by accessing a data set that is stored on one or more non-transitory computer readable media and that associates each word in a vocabulary of words with a respective numeric representation of the word in the high-dimensional space, the numeric representation of the particular word;
and
providing the numeric representation of the particular word in response to the request, wherein the data set has been generated by:
obtaining a set of training data, wherein the set of training data comprises sequences of words;
training a classifier and an embedding function on the set of training data, wherein the embedding function receives a plurality of words surrounding an unknown word in a sequence of words and maps the plurality of words into a numeric representation in accordance with a set of embedding function parameters, wherein the classifier processes the numeric representation of the sequence of words to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores measure a predicted likelihood that the corresponding word is the unknown word, and wherein training the embedding function comprises determining trained values of the embedding function parameters;
processing each word in the vocabulary using the embedding function in accordance with the trained values of the embedding function parameters to generate a respective numeric representation of each word in the vocabulary in the high-dimensional space;
associating each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space;
and
storing the data set on the one or more non-transitory computer storage media.

7 more elements

Dependencies are inferred from the claim text.

Description6,842 words

Summary1,485 words

#This specification relates to computing numeric representations of words.

#In order to predict one or more unknown words in a sequence of words, e.g., in a sentence or a phrase, some systems convert one or more of the words in the sequence into numeric representations.

#In general, one innovative aspect of the subject matter described in this specification can be embodied in a system that includes a classifier implemented in one or more computers, comprising: an embedding function layer configured to receive an input comprising a plurality of words that surround an unknown word in a sequence of words and map the plurality of words into a numeric representation in a high-dimensional space; and a classifier layer configured to process the numeric representation of the input to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word is the unknown word; and instructions to process each word in a vocabulary of words using the embedding function layer to obtain a respective numeric representation of each word in the vocabulary in the high-dimensional space and to associate each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space.

Brief description of the drawings5 figures

#FIG. 1 is a block diagram of an example word prediction system.

#FIG. 2 is a flow diagram of an example process for predicting a word based on surrounding words.

#FIG. 3 is a block diagram of another example word prediction system.

Detailed description5,270 words

#Like reference numbers and designations in the various drawings indicate like elements.

#This specification generally describes systems that can be used to generate numeric representations of words in a high-dimensional space. The numeric representations are continuous high-dimensional representations, i.e., words are represented by floating point numbers in a high-dimensional space, e.g., as high-dimensional vectors of floating point numbers. The systems can be trained so that positions of the representations in the high-dimensional space generated by the systems reflect semantic and syntactic similarities between the words they represent.

#FIG. 1 is a block diagram of an example word prediction system 100. The word prediction system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

Facts

Application

15/682,374Filed 21 Aug 2017 · continuation

Priority

15 Jan 2013

Granted

26 Mar 2019

Term ends

15 Mar 2033

Estimated · how

20 years from earliest known nonprovisional domestic filing plus recorded patent term adjustment. Maintenance-fee lapses, terminal disclaimers and statutory extensions have not been checked. The continuity chain may be incomplete; this is not an enforceability determination.

Classification7 codes

G06F 40/279
Natural language analysis: recognition of textual entities
G06F 40/30
Handling natural language data: semantic analysis
G06N 20/00
Machine learning
G10L 15/06
Speech recognition: creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the…

Family

Application61/752,911Us provisional application · filed 15 Jan 2013

Application13/841,640Parent of this continuation · filed 15 Mar 2013

Application14/715,421Parent of this continuation · filed 18 May 2015

US 9,037,464 B1Parent of this continuation · filed 19 May 2015

US 9,740,680 B1Parent of this continuation

Application14/715,421Parent of this continuation

US 10,241,997 B1This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

15 Jan 2013Priority claimedParent application 61/752,911
21 Aug 2017Application filedUS 15/682,374
22 Aug 2017Assigned to Google Inc.assignment of assignor's interest
20 Oct 2017Assigned to Google LLCchange of name
26 Mar 2019GrantedUS 10,241,997 B1
12 Oct 2026TodayWithin its term
15 Mar 2033Term endsEstimated

Computing numeric representations of words in a high-dimensional space

Life of this patent

Computing numeric representations of words in a high-dimensional space

Life of this patent

Life of this patent

Abstract

Drawings

Drawings

Claims18

Description6,842 words

Drawings

Classification7 codes

Life of this patent

Description6,842 words