# US 2021/0065012 A1 · Noisy neural network layers with noise parameters

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0065012 A1
- Document ID: US20210065012A1
- Status: Published application
- Kind: A1
- Application number: 17/020,248
- Filed: 2020-09-14
- Published: 2021-03-04
- Earliest claimed: 2017-05-20
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Mohammad Gheshlaghi Azar; Meire Fortunato; Bilal Piot; Olivier Claude Pietquin; Jacob Lee Menick; Volodymyr Mnih; Charles Blundell; Remi Munos
- Web page: https://patentgrep.com/patent/US20210065012A1

## CPC

- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/092: Reinforcement learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for selecting an action to be performed by a reinforcement learning agent. The method includes obtaining an observation characterizing a current state of an environment. For each layer parameter of each noisy layer of a neural network, a respective noise value is determined. For each layer parameter of each noisy layer, a noisy current value for the layer parameter is determined from a current value of the layer parameter, a current value of a corresponding noise parameter, and the noise value. A network input including the observation is processed using the neural network in accordance with the noisy current values to generate a network output for the network input. An action is selected from a set of possible actions to be performed by the agent in response to the observation using the network output.

## Claims

### Claim 1

A method of training a neural network,
wherein the neural network is configured to receive a network input and to process the network input to generate a network output,
wherein the neural network comprises a plurality of layers each having respective layer parameters,
wherein one or more of the layers are noisy layers, and
wherein the method comprises:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, (ii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining a training network input;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution, comprising:
for each noisy layer, sampling a predetermined number of random values from one or more predetermined probability distributions; and
for each layer parameter of each noisy layer, generating the noise value for the layer parameter by combining two or more of the sampled random values for the noisy layer;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value;
processing the training input using the neural network in accordance with the noisy current values to generate a network output for the training input;
determining a gradient of an objective function that depends on the network output with respect to the current values of the layer parameters and the current values of the noise parameters; and
determining an update to the current values of the layer parameters and the current values of the noise parameters from the gradient.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value comprises:
generating a noise modifier based on the current value of the corresponding noise parameter and the noise value; and
applying the noise modifier to the current value of the layer parameter to determine the noisy current value for the layer parameter.

### Claim 3 (depends on 2)

The method of claim 2, wherein applying the noise modifier to the current value of the layer parameter to determine the noisy current value for the layer parameter comprises:
adding the noise modifier and the current value of the layer parameter to generate the noisy current value for the layer parameter.

### Claim 4 (depends on 1)

The method of claim 1, further comprising selecting an action to be performed by a reinforcement learning agent interacting with an environment based on the network output.

### Claim 5 (depends on 4)

The method of claim 4, wherein the network input comprises an observation characterizing a state of the environment and an action from a set of actions, and wherein the network output is an estimate of a return received if the reinforcement learning agent performs the action in response to the observation.

### Claim 6 (depends on 4)

The method of claim 4, wherein the network input comprises an observation characterizing a state of the environment and the network output defines a likelihood distribution over actions in a set of possible actions to be performed by the agent in response to the observation.

### Claim 7 (depends on 1)

The method of claim 1, wherein the plurality of layers also includes one or more layers that are not noisy layers, and wherein processing the training input using the neural network comprises processing the training input in accordance with the noisy current values and the current values of the layer parameters of the layers that are not noise layers to generate the network output for the training input.

### Claim 8 (depends on 1)

The method of claim 1, wherein the one or more noisy layers include one or more fully-connected layers.

### Claim 9 (depends on 1)

The method of claim 1, wherein the one or more noisy layers include one or more convolutional layers.

### Claim 10 (depends on 1)

The method of claim 1, wherein the one or more noisy layers include one or more recurrent neural network layers.

### Claim 11 (depends on 1)

The method of claim 1, wherein the objective function additionally depends on a target output for the training network input.

### Claim 12

A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network,
wherein the neural network is configured to receive a network input and to process the network input to generate a network output,
wherein the neural network comprises a plurality of layers each having respective layer parameters,
wherein one or more of the layers are noisy layers, and
wherein the operations comprise:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, (ii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining a training network input;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution, comprising:
for each noisy layer, sampling a predetermined number of random values from one or more predetermined probability distributions; and
for each layer parameter of each noisy layer, generating the noise value for the layer parameter by combining two or more of the sampled random values for the noisy layer;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value;
processing the training input using the neural network in accordance with the noisy current values to generate a network output for the training input;
determining a gradient of an objective function that depends on the network output with respect to the current values of the layer parameters and the current values of the noise parameters; and
determining an update to the current values of the layer parameters and the current values of the noise parameters from the gradient.

### Claim 13

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network,
wherein the neural network is configured to receive a network input and to process the network input to generate a network output,
wherein the neural network comprises a plurality of layers each having respective layer parameters,
wherein one or more of the layers are noisy layers, and
wherein the operations comprise:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, (ii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining a training network input;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution, comprising:
for each noisy layer, sampling a predetermined number of random values from one or more predetermined probability distributions; and
for each layer parameter of each noisy layer, generating the noise value for the layer parameter by combining two or more of the sampled random values for the noisy layer;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value;
processing the training input using the neural network in accordance with the noisy current values to generate a network output for the training input;
determining a gradient of an objective function that depends on the network output with respect to the current values of the layer parameters and the current values of the noise parameters; and
determining an update to the current values of the layer parameters and the current values of the noise parameters from the gradient.

### Claim 14

A method of selecting an action to be performed by a reinforcement learning agent interacting with an environment using a neural network, the neural network having a plurality of layers, each layer having a plurality of layer parameters, and the plurality of layers including one or more noisy layers, the method comprising:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, (ii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining an observation characterizing a current state of the environment;
generating a network input for the neural network comprising the observation;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution, comprising:
for each noisy layer, sampling a predetermined number of random values from one or more predetermined probability distributions; and
for each layer parameter of each noisy layer, generating the noise value for the layer parameter by combining two or more of the sampled random values for the noisy layer;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value;
processing the network input using the neural network in accordance with the noisy current values to generate a network output for the network input; and
selecting an action from a set of possible actions to be performed by the agent in response to the observation using the network output;
wherein for each noisy layer, the layer parameters and the corresponding noise parameters have been trained to maximize a cumulative measure of rewards received by selecting actions to be performed by the reinforcement learning agent using network outputs generated by the neural network.

### Claim 15 (depends on 14)

The method of claim 14, wherein the network input further comprises a given action from the set of possible actions, and wherein the network output is an estimate of a return received if the reinforcement learning agent performs the given action in response to the observation.

### Claim 16 (depends on 14)

The method of claim 14, wherein the network output defines a likelihood distribution over the actions in the set of possible actions.

### Claim 17 (depends on 14)

The method of claim 14, wherein the network output identifies the action from the set of possible actions.

### Claim 18

A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for selecting an action to be performed by a reinforcement learning agent interacting with an environment using a neural network, the neural network having a plurality of layers, each layer having a plurality of layer parameters, and the plurality of layers including one or more noisy layers, the operations comprising:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, (ii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining an observation characterizing a current state of the environment;
generating a network input for the neural network comprising the observation;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution, comprising:
for each noisy layer, sampling a predetermined number of random values from one or more predetermined probability distributions; and
for each layer parameter of each noisy layer, generating the noise value for the layer parameter by combining two or more of the sampled random values for the noisy layer;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value;
processing the network input using the neural network in accordance with the noisy current values to generate a network output for the network input; and
selecting an action from a set of possible actions to be performed by the agent in response to the observation using the network output;
wherein for each noisy layer, the layer parameters and the corresponding noise parameters have been trained to maximize a cumulative measure of rewards received by selecting actions to be performed by the reinforcement learning agent using network outputs generated by the neural network.

### Claim 19

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting an action to be performed by a reinforcement learning agent interacting with an environment using a neural network, the neural network having a plurality of layers, each layer having a plurality of layer parameters, and the plurality of layers including one or more noisy layers, the operations comprising:
maintaining data specifying: (i) current values of each layer parameter of each of the plurality of layers and, (ii) for each layer parameter of each noisy layer, a current value of a corresponding noise parameter for the layer parameter in addition to the current value of the layer parameter;
obtaining an observation characterizing a current state of the environment;
generating a network input for the neural network comprising the observation;
determining, for each layer parameter of each noisy layer, a respective noise value based at least in part on a respective random value that is sampled in accordance with a probability distribution, comprising:
for each noisy layer, sampling a predetermined number of random values from one or more predetermined probability distributions; and
for each layer parameter of each noisy layer, generating the noise value for the layer parameter by combining two or more of the sampled random values for the noisy layer;
determining, for each layer parameter of each noisy layer, a noisy current value for the layer parameter from: (i) the current value of the layer parameter, (ii) the current value of the corresponding noise parameter, and (iii) the noise value;
processing the network input using the neural network in accordance with the noisy current values to generate a network output for the network input; and
selecting an action from a set of possible actions to be performed by the agent in response to the observation using the network output;
wherein for each noisy layer, the layer parameters and the corresponding noise parameters have been trained to maximize a cumulative measure of rewards received by selecting actions to be performed by the reinforcement learning agent using network outputs generated by the neural network.

### Claim 20 (depends on 19)

The non-transitory computer storage media of claim 19, wherein the network input further comprises a given action from the set of possible actions, and wherein the network output is an estimate of a return received if the reinforcement learning agent performs the given action in response to the observation.
