# US 10,503,837 B1 · Translating terms using numeric representations > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,837 B1 - Document ID: US10503837B1 - Status: Granted - Kind: B1 - Application number: 15/798,085 - Filed: 2017-10-30 - Granted: 2019-12-10 - Earliest claimed: 2014-09-17 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Ilya Sutskever; Tomas Mikolov; Jeffrey Adgate Dean; Quoc V. Le - Web page: https://patentgrep.com/patent/US10503837B1 ## CPC - G06F 40/237: Lexical tools - G06F 40/00: Handling natural language data (speech analysis or synthesis, speech recognition G10L) - G06F 40/263: Language identification - G06F 40/30: Semantic analysis - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06F 40/44: Statistical methods, e.g. probability models - G06F 40/47: Machine-assisted translation, e.g. using translation memory - G06F 40/51: Translation evaluation - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G06F 40/253: Grammatical analysis; Style critique - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, for translating terms using numeric representations. One of the methods includes obtaining data that associates each term in a vocabulary of terms in a first language with a respective high-dimensional representation of the term; obtaining data that associates each term in a vocabulary of terms in a second language with a respective high-dimensional representation of the term; receiving a first language term; and determining a translation into the second language of the first language term from the high-dimensional representation of the first language term and the high-dimensional representations of terms in the vocabulary of terms in the second language. ## Claims ### Claim 1 A method comprising: obtaining a plurality of training terms in a first language; identifying respective high-dimensional representations of the plurality of training terms; obtaining translations for the plurality of training terms, the translations being in a second language; identifying respective high-dimensional representations of the translations for the plurality of training terms; determining, by applying a machine learning technique to the high-dimensional representations of the training terms and the high-dimensional representations of the translations, trained values of transformation parameters that are used to transform a high-dimensional representation of a first term in the first language to a high-dimensional representation of a second term in the second language that is a translation of the first term, comprising: obtaining current values of the transformation parameters, processing, using the current values of the transformation parameters, each of the high-dimensional representations of one or more of the training terms to generate a respective high-dimensional representation of an output term for the training term, determining, for each of the one or more training terms, a respective difference between the high-dimensional representation of the translation for the training term and the high-dimensional representation of the output term for the training term, and modifying the current values of the transformation parameters to lower the respective differences; and providing the trained values of the transformation parameters for use in translating a term in the first language to a term in the second language using a high-dimensional representation of the term in the first language and a high-dimensional representation of the term in the second language. ### Claim 2 (depends on 1) The method of claim 1, further comprising: obtaining a plurality of terms in the first language; and processing each term of the plurality of terms in the first language in accordance with the trained values of the transformation parameters to generate a respective translation for the term, the respective translation being in the second language. ### Claim 3 (depends on 1) The method of claim 1, wherein the high-dimensional representations of the plurality of training terms are numeric representations of the plurality of training term in a high-dimensional space, and wherein the high-dimensional representations of the translations for the plurality of training terms are numeric representations of the translations for the plurality of training terms in the high-dimensional space. ### Claim 4 (depends on 1) The method of claim 1, wherein the machine learning technique is stochastic gradient descent. ### Claim 5 (depends on 1) The method of claim 1, wherein obtaining the plurality of training terms in the first language includes: obtaining a pre-determined number of most frequently occurring terms from a vocabulary of terms in the first language. ### Claim 6 A method comprising: obtaining a plurality of training terms in a first language; identifying respective high-dimensional representations of the plurality of training terms; obtaining translations for the plurality of training terms, the translations being in a second language; identifying respective high-dimensional representations of the translations for the plurality of training terms; determining, by applying a machine learning technique to the high-dimensional representations of the training terms and the high-dimensional representations of the translations, trained values of transformation parameters that are used to transform a high-dimensional representation of a first term in the first language to a high-dimensional representation of a second term in the second language that is a translation of the first term, comprising: performing multiple iterations of a machine learning training procedure to determine a matrix of values W of the transformation parameters that satisfies: min W ⁢ ∑ i = 1 n ⁢   Wx i - z i   2 , wherein n is the total number of training terms, xi is the high-dimensional representation of an i-th training term, and zi is the high-dimensional representation of the translation of the i-th training term; and providing the trained values of the transformation parameters for use in translating a term in the first language to a term in the second language using a high-dimensional representation of the term in the first language and a high-dimensional representation of the term in the second language. ### Claim 7 A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform system operations comprising: obtaining a plurality of training terms in a first language; identifying respective high-dimensional representations of the plurality of training terms; obtaining translations for the plurality of training terms, the translations being in a second language; identifying respective high-dimensional representations of the translations for the plurality of training terms; determining, by applying a machine learning technique to the high-dimensional representations of the training terms and the high-dimensional representations of the translations, trained values of transformation parameters that are used to transform a high-dimensional representation of a first term in the first language to a high-dimensional representation of a second term in the second language that is a translation of the first term, comprising: obtaining current values of the transformation parameters, processing, using the current values of the transformation parameters, each of the high-dimensional representations of one or more of the training terms to generate a respective high-dimensional representation of an output term for the training term, determining, for each of the one or more training terms, a respective difference between the high-dimensional representation of the translation for the training term and the high-dimensional representation of the output term for the training term, and modifying the current values of the transformation parameters to lower the respective differences; and providing the trained values of the transformation parameters for use in translating a term in the first language to a term in the second language using a high-dimensional representation of the term in the first language and a high-dimensional representation of the term in the second language. ### Claim 8 (depends on 7) The system of claim 7, the operations further comprising: obtaining a plurality of terms in the first language; and processing each term of the plurality of terms in the first language in accordance with the trained values of the transformation parameters to generate a respective translation for the term, the respective translation being in the second language. ### Claim 9 (depends on 7) The system of claim 7, wherein the high-dimensional representations of the plurality of training terms are numeric representations of the plurality of training term in a high-dimensional space, and wherein the high-dimensional representations of the translations for the plurality of training terms are numeric representations of the translations for the plurality of training terms in the high-dimensional space. ### Claim 10 (depends on 7) The system of claim 7, wherein the machine learning technique is stochastic gradient descent. ### Claim 11 (depends on 7) The system of claim 7, wherein obtaining the plurality of training terms in the first language includes: obtaining a pre-determined number of most frequently occurring terms from a vocabulary of terms in the first language. ### Claim 12 A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising: obtaining a plurality of training terms in a first language; identifying respective high-dimensional representations of the plurality of training terms; obtaining translations for the plurality of training terms, the translations being in a second language; identifying respective high-dimensional representations of the translations for the plurality of training terms; determining, by applying a machine learning technique to the high-dimensional representations of the training terms and the high-dimensional representations of the translations, trained values of transformation parameters that are used to transform a high-dimensional representation of a first term in the first language to a high-dimensional representation of a second term in the second language that is a translation of the first term, comprising: performing multiple iterations of a machine learning training procedure to determine a matrix of values W of the transformation parameters that satisfies: min W ⁢ ∑ i = 1 n ⁢   Wx i - z i   2 , wherein n is the total number of training terms, xi is the high-dimensional representation of an i-th training term, and zi is the high-dimensional representation of the translation of the i-th training term; and providing the trained values of the transformation parameters for use in translating a term in the first language to a term in the second language using a high-dimensional representation of the term in the first language and a high-dimensional representation of the term in the second language. ### Claim 13 A non-transitory computer-readable storage medium encoded with a computer program, the computer program comprising instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: obtaining a plurality of training terms in a first language; identifying respective high-dimensional representations of the plurality of training terms; obtaining translations for the plurality of training terms, the translations being in a second language; identifying respective high-dimensional representations of the translations for the plurality of training terms; determining, by applying a machine learning technique to the high-dimensional representations of the training terms and the high-dimensional representations of the translations, trained values of transformation parameters that are used to transform a high-dimensional representation of a first term in the first language to a high-dimensional representation of a second term in the second language that is a translation of the first term, comprising: obtaining current values of the transformation parameters, processing, using the current values of the transformation parameters, each of the high-dimensional representations of one or more of the training terms to generate a respective high-dimensional representation of an output term for the training term, determining, for each of the one or more training terms, a respective difference between the high-dimensional representation of the translation for the training term and the high-dimensional representation of the output term for the training term, and modifying the current values of the transformation parameters to lower the respective differences; and providing the trained values of the transformation parameters for use in translating a term in the first language to a term in the second language using a high-dimensional representation of the term in the first language and a high-dimensional representation of the term in the second language. ### Claim 14 (depends on 13) The computer-readable storage medium of claim 13, wherein determining the trained values of the transformation parameters comprises: obtaining initial values of the transformation parameters, processing, using the initial values of the transformation parameters, each of the high-dimensional representations of the training terms to generate a respective high-dimensional representation of an output term for the training term, determining a difference between the high-dimensional representation of the translation for the training term and the high-dimensional representation of the output term for the training term, and modifying the initial values of the transformation parameters to lower the difference. ### Claim 15 (depends on 13) The computer-readable storage medium of claim 13, further comprising: obtaining a plurality of terms in the first language; and processing each term of the plurality of terms in the first language in accordance with the trained values of the transformation parameters to generate a respective translation for the term, the respective translation being in the second language. ### Claim 16 (depends on 13) The computer-readable storage medium of claim 13, wherein the high-dimensional representations of the plurality of training terms are numeric representations of the plurality of training term in a high-dimensional space, and wherein the high-dimensional representations of the translations for the plurality of training terms are numeric representations of the translations for the plurality of training terms in the high-dimensional space. ### Claim 17 (depends on 13) The computer-readable storage medium of claim 13, wherein obtaining the plurality of training terms in the first language includes: obtaining a pre-determined number of most frequently occurring terms from a vocabulary of terms in the first language. ### Claim 18 A non-transitory computer-readable storage medium encoded with a computer program, the computer program comprising instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: obtaining a plurality of training terms in a first language; identifying respective high-dimensional representations of the plurality of training terms; obtaining translations for the plurality of training terms, the translations being in a second language; identifying respective high-dimensional representations of the translations for the plurality of training terms; determining, by applying a machine learning technique to the high-dimensional representations of the training terms and the high-dimensional representations of the translations, trained values of transformation parameters that are used to transform a high-dimensional representation of a first term in the first language to a high-dimensional representation of a second term in the second language that is a translation of the first term, comprising: performing multiple iterations of a machine learning training procedure to determine a matrix of values W of the transformation parameters that satisfies: min W ⁢ ∑ i = 1 n ⁢   Wx i - z i   2 , wherein n is the total number of training terms, xi is the high-dimensional representation of an i-th training term, and zi is the high-dimensional representation of the translation of the i-th training term; and providing the trained values of the transformation parameters for use in translating a term in the first language to a term in the second language using a high-dimensional representation of the term in the first language and a high-dimensional representation of the term in the second language.