# US 2019/0097912 A1 · Weight initialization for random neural network reinforcement learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0097912 A1
- Document ID: US20190097912A1
- Status: Published application
- Kind: A1
- Application number: 15/718,901
- Filed: 2017-09-28
- Published: 2019-03-28
- Earliest claimed: 2017-09-28
- Assignees: Cisco Technology, Inc.
- Applicants: Cisco Technology, Inc.
- Inventors: Samer Salam
- Web page: https://patentgrep.com/patent/US20190097912A1

## CPC

- H04L 45/08: {Learning-based routing, e.g. using neural networks or artificial intelligence}
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/08: Learning methods
- G06N 3/092: Reinforcement learning
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A plurality of paths through a network are determined for transmitting a packet from a source device to a destination device. The paths are modelled as nodes in a Random Neural Network, each node corresponding to a path and a reward is calculated for each of the nodes. An excitatory weight and an inhibitory weight are determined for each of the nodes in the Random Neural Network. The excitatory weight is set directly proportional to the reward corresponding to the node for which the excitatory weight is being determined, and the inhibitory weight is set inversely proportional to the reward corresponding to the node for which the inhibitory weight is being determined. A potential is determined for each of the nodes based upon the excitatory and inhibitory weights. A path corresponding to the node with the highest potential is selected, and the packet is transmitted over the selected path.

## Claims

### Claim 1

A method comprising:
determining a plurality of paths from a source network connected device to a destination network connected device through a network for transmitting the packet from a source network connected device to the destination network connected device;
modeling the plurality of paths as a plurality of nodes in a Random Neural Network, wherein each of the plurality of nodes corresponds to one of the plurality of paths;
calculating a reward for each of the plurality of nodes;
determining an excitatory weight and an inhibitory weight for each of the plurality of nodes in the Random Neural Network, wherein determining the excitatory weight comprises setting the excitatory weight for a node of the plurality of nodes for which the excitatory weight is being determined directly proportional to the reward corresponding to the node of the plurality of nodes for which the excitatory weight is being determined, and wherein determining the inhibitory weight comprises setting the inhibitory weight for a node of the plurality of nodes for which the inhibitory weight is being determined inversely proportional to the reward corresponding to the node of the plurality of nodes for which the inhibitory weight is being determined;
determining a potential for each of the plurality of nodes based upon the excitatory weights and the inhibitory weights for the plurality of nodes;
selecting a path of the plurality of paths corresponding to the node with a highest potential; and
transmitting the packet from the source network connected device to the destination network connected device over the path of the plurality of paths corresponding to the node with the highest potential.

### Claim 2 (depends on 1)

The method of claim 1, further comprising updating the excitatory weights and the inhibitory weights of the node with the highest potential in response to the transmission of the packet from the source network connected device to the destination network connected device over the path of the plurality of paths corresponding to the node with the highest potential.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
determining a second potential for each of the plurality of nodes based upon the excitatory weights and the inhibitory weights for the plurality of nodes after the updating the excitatory weights and the inhibitory weights;
selecting a path of the plurality of paths corresponding to the node with the highest second potential for transmission of a second packet from the source network connected device to the destination network connected device; and
transmitting the second packet from the source network connected device to the destination network connected device over the path of the plurality of paths corresponding to the node with the highest second potential.

### Claim 4 (depends on 1)

The method of claim 1, wherein calculating the reward comprises calculating the reward based upon a network parameter for the path corresponding to the node for which the reward is being calculated.

### Claim 5 (depends on 4)

The method of claim 4, wherein the network parameter comprises at least one of latency, jitter, a packet loss ratio, bandwidth or a path length.

### Claim 6 (depends on 1)

The method of claim 1, wherein the excitatory weight comprises an initial excitatory weight and the inhibitory weight comprises an initial inhibitory weight.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the inhibitory weight for each of the plurality of nodes in the Random Neural Network comprises determining the inhibitory weight for each of the plurality of nodes such that a sum of the inhibitory weights does not equal to zero.

### Claim 8

An apparatus comprising:
a network interface configured to communicate over a network; and
one or more processors, wherein the one or more processors are configured to:
determine a plurality of paths from a source network connected device to a destination network connected device through the network for transmitting a packet from the source network connected device to the destination network connected device;
model the plurality of paths as a plurality of nodes in a Random Neural Network, wherein each of the plurality of nodes corresponds to one of the plurality of paths;
calculate a reward for each of the plurality of nodes;
determine an excitatory weight and an inhibitory weight for each of the plurality of nodes in the Random Neural Network, wherein determining the excitatory weight comprises setting the excitatory weight for a node of the plurality of nodes for which the excitatory weight is being determined directly proportional to the reward corresponding to the node of the plurality of nodes for which the excitatory weight is being determined, and wherein determining the inhibitory weight comprises setting the inhibitory weight for a node of the plurality of nodes for which the inhibitory weight is being determined inversely proportional to the reward corresponding to the node of the plurality of nodes for which the inhibitory weight is being determined;
determine a potential for each of the plurality of nodes based upon the excitatory weights and the inhibitory weights for the plurality of nodes;
selecting a path of the plurality of paths corresponding to the node with a highest potential; and
transmit, via the network interface, the packet from the source network connected device to the destination network connected device over the path of the plurality of paths corresponding to the node with the highest potential.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the one or more processors are further configured to update the excitatory weights and the inhibitory weights of the node with the highest potential in response to the transmission of the packet from the source network connected device to the destination network connected device over the path of the plurality of paths corresponding to the node with the highest potential.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the one or more processors are further configured to:
determine a second potential for each of the plurality of nodes based upon the excitatory weights and the inhibitory weights for the plurality of nodes after the updating the excitatory weights and the inhibitory weights;
select a path of the plurality of paths corresponding to the node with the highest second potential for transmission of a second packet from the source network connected device to the destination network connected device; and
transmit the second packet from the source network connected device to the destination network connected device over the path of the plurality of paths corresponding to the node with the highest second potential.

### Claim 11 (depends on 8)

The apparatus of claim 8, wherein the one or more processors are configured to calculate the reward by calculating the reward based upon a network parameter for the path corresponding to the node for which the reward is being calculated.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the network parameter comprises at least one of latency, jitter, a packet loss ratio, bandwidth or a path length.

### Claim 13 (depends on 8)

The apparatus of claim 8, wherein the excitatory weight comprises an initial excitatory weight and the inhibitory weight comprises an initial inhibitory weight.

### Claim 14 (depends on 8)

The apparatus of claim 8, wherein the one or more processors are configured to determine the inhibitory weight for each of the plurality of nodes in the Random Neural Network by determining the inhibitory weight for each of the plurality of nodes such that a sum of the inhibitory weights does not equal to zero.

### Claim 15

A non-transitory computer readable storage medium encoded with instructions, wherein the instructions, when executed, are operable to:
determine a plurality of paths from a source network connected device to a destination network connected device through a network for transmitting a packet from the source network connected device to the destination network connected device;
model the plurality of paths as a plurality of nodes in a Random Neural Network, wherein each of the plurality of nodes corresponds to one of the plurality of paths;
calculate a reward for each of the nodes of the plurality of nodes;
determine an excitatory weight and an inhibitory weight for each of the plurality of nodes in the Random Neural Network, wherein determining the excitatory weight comprises setting the excitatory weight for a nodes of the plurality of nodes for which the excitatory weight is being determined directly proportional to the reward corresponding to the node of the plurality of nodes for which the excitatory weight is being determined, and wherein determining the inhibitory weight comprises setting the inhibitory weight for a node of the plurality of nodes for which the inhibitory weight is being determined inversely proportional to the reward corresponding to the node of the plurality of nodes for which the inhibitory weight is being determined;
determine a potential for each of the plurality of nodes based upon the excitatory weights and the inhibitory weights for the plurality of nodes;
select a path of the plurality of paths corresponding to the node with a highest potential; and
transmit the packet from the source network connected device to the destination network connected device over the path of the plurality of paths corresponding to the node with the highest potential.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium of claim 15, further comprising instructions, when executed, operable to update the excitatory weights and the inhibitory weights of the node with the highest potential in response to the transmission of the packet from the source network connected device to the destination network connected device over the path of the plurality of paths corresponding to the node with the highest potential.

### Claim 17 (depends on 16)

The non-transitory computer readable storage medium of claim 16, further comprising instructions, when executed, operable to:
determine a second potential for each of the plurality of nodes based upon the excitatory weights and the inhibitory weights for the plurality of nodes after the updating the excitatory weights and the inhibitory weights;
select a path of the plurality of paths corresponding to the node with the highest second potential for transmission of a second packet from the source network connected device to the destination network connected device; and
transmit the second packet from the source network connected device to the destination network connected device over the path of the plurality of paths corresponding to the node with the highest second potential.

### Claim 18 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the instructions operable to calculate the reward are operable to calculate the reward based upon a network parameter for the path corresponding to the node for which the reward is being calculated.

### Claim 19 (depends on 18)

The non-transitory computer readable storage medium of claim 18, wherein the network parameter comprises at least one of latency, jitter, a packet loss ratio, bandwidth or a path length.

### Claim 20 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the instructions operable to determine the inhibitory weight for each of the plurality of nodes in the Random Neural Network are operable to determine the inhibitory weight for each of the plurality of nodes such that a sum of the inhibitory weights does not equal to zero.
