# US 11,113,480 B2 · Neural machine translation systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,480 B2
- Document ID: US11113480B2
- Status: Granted
- Kind: B2
- Application number: 16/336,870
- Filed: 2017-09-25
- Published: 2020-01-30
- Granted: 2021-09-07
- Earliest claimed: 2016-09-26
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Mohammad Norouzi; Zhifeng Chen; Yonghui Wu; Michael Schuster; Quoc V. Le
- Web page: https://patentgrep.com/patent/US11113480B2

## CPC

- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/44: Statistical methods, e.g. probability models
- G06F 40/56: Natural language generation
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural machine translation. One of the systems includes an encoder neural network comprising: an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token, an input backward LSTM layer configured to process each input token in a backward order to generate a respective backward representation of each input token and a plurality of hidden LSTM layers configured to process a respective combined representation of each of the input tokens in the forward order to generate a respective encoded representation of each of the input tokens; and a decoder subsystem configured to receive the respective encoded representations and to process the encoded representations to generate an output sequence.

## Claims

### Claim 1

A neural machine translation system implemented by one or more computers, the neural machine translation system configured to receive an input sequence of input tokens representing a first sequence of words in a first natural language and to generate an output sequence of output tokens representing a second sequence of words that is a translation of the first sequence into a second natural language, the neural machine translation system comprising:
an encoder neural network comprising:
an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token,
an input backward LSTM layer configured to process each input token in the input sequence in a backward order to generate a respective backward representation of each input token,
a combining layer configured to, for each input token in the input sequence, combine the forward representation of the input token and the backward representation of the input token to generate a combined representation of the input token, and
a plurality of hidden LSTM layers configured to process each combined representation generated by the combining layer in the forward order to generate a respective encoded representation of each of the input tokens in the input sequence, wherein the plurality of hidden LSTM layers are unidirectional and arranged in a stack one after the other; and
a decoder subsystem configured to receive the respective encoded representation of each of the input tokens in the input sequence and to process the encoded representations to generate the output sequence.

### Claim 2 (depends on 1)

The neural machine translation system of claim 1, wherein the plurality of hidden LSTM layers comprises at least seven LSTM layers.

### Claim 3 (depends on 2)

The neural machine translation system of claim 2, wherein at least two of the plurality of hidden LSTM layers are connected with a residual connection.

### Claim 4 (depends on 1)

The neural machine translation system of claim 1, wherein the decoder subsystem comprises:
a decoder neural network, the decoder neural network comprising:
a plurality of LSTM layers arranged in a stack one on top of the other and configured to, for each of a plurality of positions in the output sequence:
receive an attention context vector and the output token at a preceding position in the output sequence, and
process the attention context vector and the token at the preceding position in the output sequence to generate an LSTM output for the position, and
a softmax output layer, the softmax output layer configured to, for each of the plurality of positions, receive the LSTM output for the position and to generate a respective score for each output token in a vocabulary of output tokens.

### Claim 5 (depends on 4)

The neural machine translation system of claim 4, wherein the plurality of LSTM layers comprises at least eight LSTM layers.

### Claim 6 (depends on 4)

The neural machine translation system of any one of claim 4, wherein at least two of the plurality of LSTM layers are connected with a residual connection.

### Claim 7 (depends on 6)

The neural machine translation system of claim 6, wherein a first LSTM layer in the stack is configured to, for each of the plurality of positions:
receive a first layer input comprising the attention context vector, a layer output generated by a LSTM layer directly below the first LSTM layer in the stack, and a layer input to the LSTM layer directly below the first LSTM layer in the stack; and
process the first layer input in accordance with a current hidden state of the first LSTM layer to generate a first layer output and to update the current hidden state.

### Claim 8 (depends on 4)

The neural machine translation system of claim 4, wherein the decoder system further comprises:
an attention subsystem, wherein the attention subsystem is configured to, for each of the plurality of positions:
receive an attention input comprising a bottom layer output generated by a bottom LSTM layer in the stack of LSTM layers in the decoder neural network for the preceding position; and
process the attention input to generate the attention context vector for the time step.

### Claim 9 (depends on 8)

The neural machine translation system of claim 8, wherein processing the bottom layer output and the respective encoded representation of each of the input tokens in the input sequence to generate the attention context vector for the time step comprises, for each of the plurality of time steps:
processing the bottom layer output and the encoded representations through one or more neural network layers to generate a respective score for each of the input tokens;
generating a respective weight for each of the input tokens from the respective scores; and
determining a weighted sum of the encoded representations in accordance with the weights.

### Claim 10 (depends on 1)

The neural machine translation system of any one of claim 1, wherein the input and output tokens are wordpieces.

### Claim 11 (depends on 1)

The neural machine translation system of claim 1, wherein the encoder neural network and the decoder subsystem are configured to perform quantized computations during generation of the output sequence.

### Claim 12

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to implement a neural machine translation system that is configured to receive an input sequence of input tokens representing a first sequence of words in a first natural language and to generate an output sequence of output tokens representing a second sequence of words that is a translation of the first sequence into a second natural language, the neural machine translation system comprising:
an encoder neural network comprising:
an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token,
an input backward LSTM layer configured to process each input token in the input sequence in a backward order to generate a respective backward representation of each input token,
a combining layer configured to, for each input token in the input sequence, combine the forward representation of the input token and the backward representation of the input token to generate a combined representation of the input token, and
a plurality of hidden LSTM layers configured to process each combined representation generated by the combining layer in the forward order to generate a respective encoded representation of each of the input tokens in the input sequence, wherein the plurality of hidden LSTM layers are unidirectional and arranged in a stack one after the other; and
a decoder subsystem configured to receive the respective encoded representation of each of the input tokens in the input sequence and to process the encoded representations to generate the output sequence.

### Claim 13 (depends on 12)

The computer-readable storage media of claim 12, wherein the decoder subsystem comprises:
a decoder neural network, the decoder neural network comprising:
a plurality of LSTM layers arranged in a stack one on top of the other and configured to, for each of a plurality of positions in the output sequence:
receive an attention context vector and the output token at a preceding position in the output sequence, and
process the attention context vector and the token at the preceding position in the output sequence to generate an LSTM output for the position, and
a softmax output layer, the softmax output layer configured to, for each of the plurality of positions, receive the LSTM output for the position and to generate a respective score for each output token in a vocabulary of output tokens.

### Claim 14 (depends on 13)

The computer-readable storage media of claim 13, wherein the decoder system further comprises:
an attention subsystem, wherein the attention subsystem is configured to, for each of the plurality of positions:
receive an attention input comprising a bottom layer output generated by a bottom LSTM layer in the stack of LSTM layers in the decoder neural network for the preceding position; and
process the attention input to generate the attention context vector for the time step.

### Claim 15

A method performed by one or more computers, the method comprising:
receiving an input sequence of input tokens representing a first sequence of words in a first natural language, and
processing the input sequence using a neural machine translation system to generate an output sequence of output tokens representing a second sequence of words that is a translation of the first sequence into a second natural language, the neural machine translation system comprising:
an encoder neural network comprising:
an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token,
an input backward LSTM layer configured to process each input token in the input sequence in a backward order to generate a respective backward representation of each input token,
a combining layer configured to, for each input token in the input sequence, combine the forward representation of the input token and the backward representation of the input token to generate a combined representation of the input token, and
a plurality of hidden LSTM layers configured to process each combined representation generated by the combining layer in the forward order to generate a respective encoded representation of each of the input tokens in the input sequence, wherein the plurality of hidden LSTM layers are unidirectional and arranged in a stack one after the other; and
a decoder subsystem configured to receive the respective encoded representation of each of the input tokens in the input sequence and to process the encoded representations to generate the output sequence.

### Claim 16 (depends on 15)

The method of claim 15, wherein the plurality of hidden LSTM layers comprises at least seven LSTM layers.

### Claim 17 (depends on 16)

The method of claim 16, wherein at least two of the plurality of hidden LSTM layers are connected with a residual connection.

### Claim 18 (depends on 15)

The method of claim 15, wherein the decoder subsystem comprises:
a decoder neural network, the decoder neural network comprising:
a plurality of LSTM layers arranged in a stack one on top of the other and configured to, for each of a plurality of positions in the output sequence;
receive an attention context vector and the output token at a preceding position in the output sequence, and
process the attention context vector and the token at the preceding position in the output sequence to generate an LSTM output for the position, and
a softmax output layer, the softmax output layer configured to, for each of the plurality of positions, receive the LSTM output for the position and to generate a respective score for each output token in a vocabulary of output tokens.

### Claim 19 (depends on 18)

The method of claim 18, wherein the plurality of LSTM layers comprises at least eight LSTM layers.