# US 10,375,585 B2 · System and method for deep learning and wireless network optimization using deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,375,585 B2
- Document ID: US10375585B2
- Status: Granted
- Kind: B2
- Application number: 15/643,266
- Filed: 2017-07-06
- Published: 2019-01-10
- Granted: 2019-08-06
- Earliest claimed: 2017-07-06
- Assignees: Futurwei Technologies, Inc.
- Applicants: Futurewei Technologies, Inc.
- Inventors: Yongxi Tan; Jin Yang; Qitao Song; Yunjun Chen; Zhangxiang Ye
- Web page: https://patentgrep.com/patent/US10375585B2

## CPC

- H04W 24/02: Arrangements for optimising operational condition
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- H04W 24/08: Testing, {supervising or monitoring} using real traffic
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A neural network is trained using deep reinforcement learning (DRL) techniques for adjusting cell parameters of a wireless network by generating a plurality of experience tuples, and updating the neural network based on the generated experience tuples. The trained neural network may be used to select actions to adjust the cell parameters. Each experience tuple includes a cell identifier, a first state, a second state, an action applied to the cell that moves the cell from the first state to the second state, a local reward, and a global reward. The neural network is updated based on whether or not each action is acceptable, which is determined based on the global reward and the local reward associated with each action.

## Claims

### Claim 1

A method comprising:
initializing a neural network with a set of weight values, the neural network being used to determine actions that adjust one or more settings of cells associated with base stations in a wireless network, each base station providing communication services to user equipments (UEs) within one or more cells; and
training the neural network by using a deep reinforcement learning (DRL) process, the DRL processing comprising:
generating a first plurality of experience tuples for a plurality of cells in the wireless network, each experience tuple comprising a cell identifier that identifies a cell, a first state of the cell, a second state of the cell, an action that causes the cell to transit from the first state to the second state, and a reward value for taking the action, wherein a state of a cell comprises a setting of a base station providing a coverage area of the cell, and a reward value is calculated using a cost function based on measurement reports received from UEs in the wireless network, wherein each experience tuple can be a DRL-generated experience tuple in which a respective action is selected by a DRL agent based on the neural network according to a DRL technique or an expert-generated experience tuple in which the respective action is provided based on expert experience, and wherein whether an action is selected by the DRL agent based on the neural network or provided based on the expert experience is determined based on a first criterion; and
selecting a second plurality of experience tuples from the first plurality of experience tuples; and
updating the set of weight values of the neural network according to reward values in the second plurality of experience tuples.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first criterion is a probability based criterion, a similarity based criterion or a threshold based criterion.

### Claim 3 (depends on 1)

The method of claim 1, wherein initializing the neural network with the set of weight values comprises:
determining whether the neural network is initialized with randomly selected weight values or weight values obtained from an expert neural network, the expert neural network having been trained using a deep learning technique and being able to provide weight values for initializing the neural network.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating the first plurality of experience tuples further comprises:
determining, based on the first criterion, whether an experience tuple is generated using an action that is selected by the DRL agent based on the neural network or whether the experience tuple is provided based on the expert experience.

### Claim 5 (depends on 1)

The method of claim 1, wherein selecting the second plurality of experience tuples from the first plurality of experience tuples comprises:
determining whether selecting a DRL-generated experience tuple or an expert-generated experience tuple from the first plurality of experience tuples based on a second criterion.

### Claim 6 (depends on 5)

The method of claim 5, wherein the second criterion is a threshold based criterion, a probability based criterion, a similarity based criterion, a criterion based on relationship among experience tuples, or a criterion based on importance sampling.

### Claim 7

A method comprising:
training a neural network for a plurality of cells in a wireless network using a deep reinforcement learning (DRL) process for adjusting one or more cell parameters of cells associated with base stations in the wireless network, each base station providing communication services to user equipments (UEs) within a coverage area of one or more cells, and the neural network being trained to determine actions that can be performed on the base stations, wherein the DRL process comprises:
generating a plurality of experience tuples for a plurality of cells in the wireless network, comprising:
generating a state tensor for each of the plurality of cells, each state tensor indicating a state of a respective cell, wherein a state of a cell comprises a setting of a base station associated with the cell, the base station providing a coverage area of the cell;
selecting an action for each of the plurality of cells, the action moving the respective cell from one state to another state, wherein an action comprises information for adjusting a setting of a base station associated with a cell;
applying respective actions selected for the plurality of cells to the respective cells to adjust one or more cell parameters; and
generating an experience tuple for each of the plurality of cells based on the respective action applied, the experience tuple comprising a cell identifier identifying the respective cell, a first state of the respective cell that is indicated by a respective state tensor, a second state of the respective cell, the action applied to the respective cell that moves the respective cell from the first state to the second state, a local reward calculated for applying the action to the respective cell, and a global reward calculated for applying the action to the respective cell, the local reward being calculated based on a local cost function and the global reward being calculated based on a global cost function; and
determining whether or not an action of a first experience tuple in the plurality of experience tuples corresponding to a first cell is acceptable based on the local reward and the global reward of the first experience tuple; and
updating weights of the neural network based on whether or not the action is acceptable;
selecting an action for adjusting a cell parameter of a cell in the wireless network based on the trained neural network; and
instructing to adjust the cell parameter of the cell in the wireless network according to the selected action.

### Claim 8 (depends on 7)

The method of claim 7, wherein a state tensor of a cell comprises at least a cell parameter of the cell and information obtained from a measurement report provided by a UE.

### Claim 9 (depends on 7)

The method of claim 7, wherein a state tensor comprises information of inter-site distance (ISD), a height of a base station, an antenna azimuth, an antenna mechanical tilt (mTilt), an antenna electronic tilt (eTilt), a key performance indicator, reference signal received power (RSRP), reference signal received quality (RSRP), signal interference to noise ratio (SINR), channel quality indicator (CQI)), an objective function, a cumulative distribution function of network performance measurements, or an interference factor matrix.

### Claim 10 (depends on 7)

The method of claim 7, wherein the one or more cell parameters comprises an engineer parameter of a base station associated with a cell.

### Claim 11 (depends on 7)

The method of claim 7, wherein the DRL process further comprises selecting the plurality of cells from the wireless network based on a predefined criterion.

### Claim 12 (depends on 7)

The method of claim 7, wherein selecting the action for each of the plurality of cells comprises selecting an action for a cell in the plurality of cells using the neural network based on the state tensor of the cell.

### Claim 13 (depends on 7)

The method of claim 7, wherein selecting the action for each of the plurality of cells comprises selecting an action for a cell in the plurality of cells according to expert experience.

### Claim 14 (depends on 7)

The method of claim 7, wherein determining whether or not the action of the first experience tuple in the plurality of experience tuples is acceptable comprises:
determining whether the global reward of the first experience tuple satisfies a first criterion;
when the global reward of the first experience tuple satisfies the first criterion, determining whether the local reward of the first experience tuple satisfies a second criterion; and
when the global reward of the first experience tuple satisfies the first criterion and the local reward of the first experience tuple satisfies the second criterion, determining that the action is acceptable.

### Claim 15 (depends on 14)

The method of claim 14, further comprising determining that the action is not acceptable when the global reward of the first experience tuple does not satisfy the first criterion or the local reward of the first experience tuple does not satisfy the second criterion.

### Claim 16 (depends on 14)

The method of claim 14, wherein determining whether the global reward of the first experience tuple satisfies the first criterion comprises:
determining that the global reward of the first experience tuple satisfies the first criterion when the global reward exceeds a first threshold.

### Claim 17 (depends on 16)

The method of claim 16, wherein the first threshold is a best global reward value that has been obtained for the first cell.

### Claim 18 (depends on 14)

The method of claim 14, wherein determining whether the global reward of the first experience tuple satisfies the first criterion comprises:
determining that the global reward of the first experience tuple does not satisfies the first criterion when the global reward is less than a second threshold.

### Claim 19 (depends on 18)

The method of claim 18, wherein the second threshold is a best global reward value that has been obtained for the first cell.

### Claim 20 (depends on 14)

The method of claim 14, wherein determining whether the local reward of the first experience tuple satisfies the second criterion comprises:
determining that the local reward of the first experience tuple satisfies the second criterion when the local reward exceeds a third threshold.

### Claim 21 (depends on 20)

The method of claim 20, wherein the third threshold is an initial local reward value obtained for the first cell.

### Claim 22 (depends on 14)

The method of claim 14, wherein determining whether the local reward of the first experience tuple satisfies the second criterion comprises:
determining that the local reward of the first experience tuple does not satisfy the second criterion when the local reward does not exceed a fourth threshold.

### Claim 23 (depends on 7)

The method of claim 7, wherein updating the weights of the neural network based on whether or not the action is acceptable comprises:
applying a positive gradient when the action is acceptable; and
applying a negative gradient when the action is not acceptable.

### Claim 24 (depends on 7)

The method of claim 7, wherein the local reward and the global reward are calculated using information from different cells in the wireless network.

### Claim 25 (depends on 7)

The method of claim 7, wherein the one or more cell parameters comprise an antenna tilt, an antenna azimuth, or an antenna transmission power.

### Claim 26

An apparatus comprising:
a non-transitory memory storage comprising instructions; and
one or more processors in communication with the memory storage, wherein the one or more processors execute the instructions to:
train a neural network for a plurality of cells in a wireless network using a deep reinforcement learning (DRL) process for adjusting one or more cell parameters of cells associated with base stations in the wireless network, each base station providing communication services to user equipments (UEs) within a coverage area of one or more cells, and the neural network being trained to determine actions that can be performed on the base stations, wherein the DRL process comprises:
generating a plurality of experience tuples for a plurality of cells in the wireless network, comprising:
generating a state tensor for each of the plurality of cells, each state tensor indicating a state of the respective cell, wherein a state of a cell comprises a setting of a base station associated with the cell, the base station providing a coverage area of the cell;
selecting an action for each of the plurality of cells, the action moving the respective cell from one state to another state, wherein an action comprises information for adjusting a setting of a base station associated with the cell;
applying respective actions selected for the plurality of cells to the respective cells to adjust one or more cell parameters; and
generating an experience tuple for each of the plurality of cells based on the respective action applied, the experience tuple comprising a cell identifier identifying the respective cell, a first state of the respective cell that is indicated by a respective state tensor, a second state of the respective cell, the action applied to the respective cell that moves the respective cell from the first state to the second state, a local reward calculated for applying the action to the respective cell, and a global reward calculated for applying the action to the respective cell, the local reward being calculated based on a local cost function and the global reward being calculated based on a global cost function; and
determining whether or not an action of a first experience tuple in the plurality of experience tuples corresponding to a first cell is acceptable based on the local reward and the global reward of the first experience tuple; and
updating weights of the neural network based on whether or not the action is acceptable; and
select an action for adjusting a cell parameter of a cell in the wireless network based on the trained neural network; and
instruct to adjust the cell parameter of the cell in the wireless network according to the selected action.

### Claim 27 (depends on 26)

The apparatus of claim 26, wherein a state tensor of a cell comprises at least a cell parameter of the cell and information obtained from a measurement report provided by a UE.

### Claim 28 (depends on 26)

The apparatus of claim 26, wherein a state tensor comprises information of inter-site distance (ISD), a height of a base station, an antenna azimuth, an antenna mechanical tilt (mTilt), an antenna electronic tilt (eTilt), a key performance indicator, reference signal received power (RSRP), reference signal received quality (RSRP), signal interference to noise ratio (SINR), channel quality indicator (CQI)), an objective function, a cumulative distribution function of network performance measurements, or an interference factor matrix.

### Claim 29 (depends on 26)

The apparatus of claim 26, wherein selecting the action for each of the plurality of cells comprises selecting an action for a cell in the plurality of cells using the neural network based on the state tensor of the cell.

### Claim 30 (depends on 26)

The apparatus of claim 26, wherein selecting the action for each of the plurality of cells comprises selecting an action for a cell in the plurality of cells according to expert experience.

### Claim 31 (depends on 26)

The apparatus of claim 26, wherein determining whether or not the action of the first experience tuple in the plurality of experience tuples is acceptable comprises:
determining whether the global reward of the first experience tuple satisfies a first criterion;
when the global reward of the first experience tuple satisfies the first criterion, determining whether the local reward of the first experience tuple satisfies a second criterion; and
when the global reward of the first experience tuple satisfies the first criterion and the local reward of the first experience tuple satisfies the second criterion, determining that the action is acceptable.

### Claim 32 (depends on 26)

The apparatus of claim 26, wherein updating the weights of the neural network based on whether or not the action is acceptable comprises:
applying a positive gradient when the action is acceptable; and
applying a negative gradient when the action is not acceptable.