US 11,900,235 B1Granted

Generating parameter values for recurrent neural networks

AssigneeGoogle Inc.

InventorsAndrew M. Dai, Quoc V. Le, David Ha

Life of this patent

Estimating the term…

Loading patent…

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using recurrent neural networks. One of the systems includes a main recurrent neural network comprising one or more recurrent neural network layers and a respective hyper recurrent neural network corresponding to each of the one or more recurrent neural network layers, wherein each hyper recurrent neural network is configured to, at each of a plurality of time steps: process the layer input at the time step to the corresponding recurrent neural network layer, the current layer hidden state of the corresponding recurrent neural network layer, and a current hypernetwork hidden state of the hyper recurrent neural network to generate an updated hypernetwork hidden state.

Claims20

Dependent claims

A method of training a main recurrent neural network comprising one or more recurrent neural network layers and respective hyper recurrent neural networks corresponding to each of the one or more recurrent neural network layers, the method comprising:

processing a sequence of network inputs having a respective network input at each of a plurality of time steps to generate a sequence of network outputs, the processing comprising, for each of the plurality of time steps:
for each of the one or more recurrent neural network layers of the main recurrent neural network:
providing a layer input for the recurrent neural network layer at the time step as input to the respective hyper recurrent neural network corresponding to the recurrent neural network layer, wherein the hyper recurrent neural network is configured to:
generate a current hypernetwork input by combining:
(1) the layer input at the time step to the corresponding recurrent neural network layer, (2) a current layer hidden state of the corresponding recurrent neural network layer, and (3) a current hypernetwork hidden state of the hyper recurrent neural network;
and
process current hypernetwork input to generate an updated hypernetwork hidden state;
determining, from the updated hypernetwork hidden state for the corresponding hyper recurrent neural network, updated values for a set of layer parameters for the recurrent neural network layer;
and
providing the layer input as input to the recurrent neural network layer, wherein the recurrent neural network layer is configured to process the layer input and a current layer hidden state in accordance with the updated values of the set of layer parameters to generate an updated layer hidden state;
and
updating parameters of the hyper recurrent neural networks based on the sequence of network outputs through backpropagation.

9 more elements

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform training of a main recurrent neural network comprising one or more recurrent neural network layers and respective hyper recurrent neural networks corresponding to each of the one or more recurrent neural network layers, the training comprising:

processing a sequence of network inputs having a respective network input at each of a plurality of time steps to generate a sequence of network outputs, the processing comprising, for each of the plurality of time steps:
for each of the one or more recurrent neural network layers of the main recurrent neural network:
providing a layer input for the recurrent neural network layer at the time step as input to the respective hyper recurrent neural network corresponding to the recurrent neural network layer, wherein the hyper recurrent neural network is configured to:
generate a current hypernetwork input by combining:
(1) the layer input at the time step to the corresponding recurrent neural network layer, (2) a current layer hidden state of the corresponding recurrent neural network layer, and (3) a current hypernetwork hidden state of the hyper recurrent neural network;
and
process current hypernetwork input to generate an updated hypernetwork hidden state;
determining, from the updated hypernetwork hidden state for the corresponding hyper recurrent neural network, updated values for a set of layer parameters for the recurrent neural network layer;
and
providing the layer input as input to the recurrent neural network layer, wherein the recurrent neural network layer is configured to process the layer input and a current layer hidden state in accordance with the updated values of the set of layer parameters to generate an updated layer hidden state;
and
updating parameters of the main recurrent neural network layers based on the sequence of network outputs through backpropagation.

9 more elements

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform training of a main recurrent neural network comprising one or more recurrent neural network layers and respective hyper recurrent neural networks corresponding to each of the one or more recurrent neural network layers, the training comprising:

processing a sequence of network inputs having a respective network input at each of a plurality of time steps to generate a sequence of network outputs, the processing comprising, for each of the plurality of time steps:
for each of the one or more recurrent neural network layers of the main recurrent neural network:
providing a layer input for the recurrent neural network layer at the time step as input to the respective hyper recurrent neural network corresponding to the recurrent neural network layer, wherein the hyper recurrent neural network is configured to:
generate a current hypernetwork input by combining:
(1) the layer input at the time step to the corresponding recurrent neural network layer, (2) a current layer hidden state of the corresponding recurrent neural network layer, and (3) a current hypernetwork hidden state of the hyper recurrent neural network;
and
process current hypernetwork input to generate an updated hypernetwork hidden state;
determining, from the updated hypernetwork hidden state for the corresponding hyper recurrent neural network, updated values for a set of layer parameters for the recurrent neural network layer;
and
providing the layer input as input to the recurrent neural network layer, wherein the recurrent neural network layer is configured to process the layer input and a current layer hidden state in accordance with the updated values of the set of layer parameters to generate an updated layer hidden state;
and
updating parameters of the main recurrent neural network layers based on the sequence of network outputs through backpropagation.

9 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/470,716Filed 9 Sep 2021 · continuation
Priority: 26 Sep 2017
Granted: 13 Feb 2024
Term ends: Not estimatedThe source records don’t support an estimate.

Classification8 codes

G06N 3/0442
Recurrent networks, e.g. Hopfield networks: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
G06N 3/084
Learning methods: backpropagation, e.g. using gradient descent
G06N 3/044
Architecture, e.g. interconnection topology: recurrent networks, e.g. Hopfield networks
G06N 3/045
Architecture, e.g. interconnection topology: combinations of networks