# US 2021/0174209 A1 · Neural network obtaining method and related device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174209 A1
- Document ID: US20210174209A1
- Status: Published application
- Kind: A1
- Application number: 17/181,810
- Filed: 2021-02-22
- Published: 2021-06-10
- Earliest claimed: 2019-09-29
- Assignees: Huawei Technologies Co., Ltd.; Tsinghua University
- Applicants: Huawei Technologies Co., Ltd.; Tsinghua University
- Inventors: Yuzheng ZHUANG; Siyuan LI; Rui WANG; Wulong LIU; Chongjie ZHANG
- Web page: https://patentgrep.com/patent/US20210174209A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 16/2379: {Updates performed during online database operations; commit processing}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/092: Reinforcement learning
- G06N 3/096: Transfer learning
- G08G 1/0116: {from roadside infrastructure, e.g. beacons}
- G08G 1/012: {from other sources than vehicle or roadside beacons, e.g. mobile networks}
- G08G 1/0129: {for creating historical data or processing based on historical data}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/063: using electronic means
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

A neural network obtaining method and a related device are provided. The method may be applied to a scenario in which reinforcement learning is performed on a neural network in the artificial intelligence field. After obtaining a first task, a server obtains a first success rate of completing the first task by using a first neural network. When the first success rate is less than a preset threshold, the server generates a second neural network and a new skill. The server trains, based on a simulated environment corresponding to the first task, the second neural network by using a reinforcement learning algorithm, until a second success rate of completing the first task by using the second neural network is greater than or equal to the preset threshold. The server stores the trained second neural network and the new skill.

## Claims

### Claim 1

A neural network obtaining method, wherein the method comprises:
obtaining a first success rate of completing a first task by using a first neural network, wherein the first neural network is used to select, from a skill library, a skill used to complete the first task;
when the first success rate is less than a preset threshold, obtaining a second neural network and a new skill, and adding the new skill to the skill library to update the skill library, wherein the second neural network is used to select, from the updated skill library, a skill used to complete the first task;
training, based on a simulated environment corresponding to the first task, the second neural network and the new skill by using a reinforcement learning algorithm, until a second success rate of completing the first task by using the second neural network is greater than or equal to the preset threshold; and
storing the trained second neural network and the updated skill library.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the training, based on a simulated environment corresponding to the first task, the second neural network by using a reinforcement learning algorithm comprises:
inputting environment status information into the second neural network, to obtain a skill selected by using the second neural network, wherein the environment status information comprises status information of an intelligent device and information about a surrounding environment of the intelligent device in the simulated environment corresponding to the first task;
obtaining a control instruction by using the skill selected by using the second neural network, to control the intelligent device to execute the first task, and obtaining data for executing the first task by the intelligent device; and
updating, based on the obtained data for executing the first task by the intelligent device, parameters of the second neural network by using the reinforcement learning algorithm.

### Claim 3 (depends on 2)

The method according to claim 2, wherein after the inputting environment status information into the second neural network, to obtain a skill selected by using the second neural network, the method further comprises:
when the skill selected by using the second neural network is the new skill, updating, based on the obtained data for executing the first task by the intelligent device, a parameter of the new skill by using a reinforcement learning algorithm.

### Claim 4 (depends on 2)

The method according to claim 2, wherein the updating parameters of the second neural network by using the reinforcement learning algorithm comprises:
updating the parameters of the second neural network by using a first reinforcement learning algorithm, wherein the first reinforcement learning algorithm is a proximal policy optimization (PPO) algorithm, a trust region policy optimization (TRPO) algorithm, or a temporal-difference (TD) algorithm.

### Claim 5 (depends on 3)

The method according to claim 3, wherein the updating a parameter of the new skill by using a reinforcement learning algorithm comprises:
updating the parameter of the new skill by using a second reinforcement learning algorithm, wherein the second reinforcement learning algorithm is a deep deterministic policy gradient DDPG algorithm, a deep Q-network DQN algorithm, a PPO algorithm, or a soft actor-critic SAC algorithm.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the method further comprises:
initializing a parameter of the second neural network by using a parameter of the first neural network, wherein the first neural network is obtained by performing, based on a simulated environment corresponding to a second task, training by using a reinforcement learning algorithm, and semantic information of the second task is similar to semantic information of the first task.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the first neural network and the skill library are obtained by performing, based on a simulated environment corresponding to a second task, training by using a reinforcement learning algorithm, and semantic information of the second task is similar to semantic information of the first task; and the method further comprises:
when the first success rate is greater than or equal to the preset threshold, determining the first neural network as a neural network used to complete the first task; and
storing the first neural network and the skill library.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the obtaining a first success rate of completing a first task by using a first neural network comprises:
inputting the environment status information into the first neural network, to obtain a skill selected by using the first neural network, wherein the environment status information comprises the status information of the intelligent device and the information about the surrounding environment of the intelligent device in the simulated environment corresponding to the first task;
obtaining a control instruction by using the skill selected by using the first neural network, to control the intelligent device to execute the first task;
repeating the foregoing steps until a quantity of times the first task is completed by using the first neural network reaches a first quantity of times;
counting a quantity of times a result of completing the first task is a success in the first quantity of times, to obtain a second quantity of times; and
calculating the first success rate based on the first quantity of times and the second quantity of times.

### Claim 9 (depends on 8)

The method according to claim 8, wherein before the inputting the environment status information into the first neural network, the method further comprises:
obtaining a control instruction by using a skill selected by using the first neural network, to control the intelligent device to execute the first task, so as to determine whether a loss function of the first neural network is converged in the simulated environment corresponding to the first task, wherein the first neural network is obtained by performing, based on a simulated environment corresponding to a second task, training by using a reinforcement learning algorithm, and semantic information of the second task is similar to semantic information of the first task; and
when the loss function of the first neural network is not converged, training, based on the simulated environment corresponding to the first task, the first neural network by using a reinforcement learning algorithm, until a convergence condition of the loss function of the first neural network is met, to obtain the mature first neural network; and
the obtaining a first success rate of completing a first task by using a first neural network comprises:
obtaining a success rate of completing the first task by using the mature first neural network, and using the success rate as the first success rate.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the obtaining a first success rate of completing a first task by using a first neural network comprises:
obtaining semantic information of the first task, and obtaining semantic information of a second task, wherein the semantic information of the second task is similar to the semantic information of the first task, and the first neural network is obtained by performing, based on a simulated environment corresponding to the second task, training by using a reinforcement learning algorithm; and
obtaining the first success rate based on a similarity between the semantic information of the first task and the semantic information of the second task, wherein a higher similarity indicates a higher first success rate.

### Claim 11 (depends on 1)

The method according to claim 1, wherein
a skill in the skill library is a driving skill;
a skill in the skill library is a carrying skill; or
a skill in the skill library is a grabbing skill.

### Claim 12

A server, comprising a processor, wherein the processor is coupled to a memory; the memory stores a program instruction, which when executed by the processor, cause the processor to perform the operations, comprising:
obtaining a first success rate of completing a first task by using a first neural network, wherein the first neural network is used to select, from a skill library, a skill used to complete the first task;
when the first success rate is less than a preset threshold, obtaining a second neural network and a new skill, and adding the new skill to the skill library to update the skill library, wherein the second neural network is used to select, from the updated skill library, a skill used to complete the first task;
training, based on a simulated environment corresponding to the first task, the second neural network and the new skill by using a reinforcement learning algorithm, until a second success rate of completing the first task by using the second neural network is greater than or equal to the preset threshold; and
storing the trained second neural network and the updated skill library.

### Claim 13 (depends on 12)

The server according to claim 12, wherein the training, based on a simulated environment corresponding to the first task, the second neural network by using a reinforcement learning algorithm comprises:
inputting environment status information into the second neural network, to obtain a skill selected by using the second neural network, wherein the environment status information comprises status information of an intelligent device and information about a surrounding environment of the intelligent device in the simulated environment corresponding to the first task;
obtaining a control instruction by using the skill selected by using the second neural network, to control the intelligent device to execute the first task, and obtaining data for executing the first task by the intelligent device; and
updating, based on the obtained data for executing the first task by the intelligent device, parameters of the second neural network by using the reinforcement learning algorithm.

### Claim 14 (depends on 13)

The server according to claim 13, wherein after the inputting environment status information into the second neural network, to obtain a skill selected by using the second neural network, the method further comprises:
when the skill selected by using the second neural network is the new skill, updating, based on the obtained data for executing the first task by the intelligent device, a parameter of the new skill by using a reinforcement learning algorithm.

### Claim 15 (depends on 13)

The server according to claim 13, wherein the updating parameters of the second neural network by using the reinforcement learning algorithm comprises:
updating the parameters of the second neural network by using a first reinforcement learning algorithm, wherein the first reinforcement learning algorithm is a proximal policy optimization (PPO) algorithm, a trust region policy optimization (TRPO) algorithm, or a temporal-difference (TD) algorithm.

### Claim 16 (depends on 14)

The server according to claim 14, wherein the updating a parameter of the new skill by using a reinforcement learning algorithm comprises:
updating the parameter of the new skill by using a second reinforcement learning algorithm, wherein the second reinforcement learning algorithm is a deep deterministic policy gradient DDPG algorithm, a deep Q-network DQN algorithm, a PPO algorithm, or a soft actor-critic SAC algorithm.

### Claim 17 (depends on 12)

The server according to claim 12, wherein the method further comprises:
initializing a parameter of the second neural network by using a parameter of the first neural network, wherein the first neural network is obtained by performing, based on a simulated environment corresponding to a second task, training by using a reinforcement learning algorithm, and semantic information of the second task is similar to semantic information of the first task.

### Claim 18 (depends on 12)

The server according to claim 12, wherein the first neural network and the skill library are obtained by performing, based on a simulated environment corresponding to a second task, training by using a reinforcement learning algorithm, and semantic information of the second task is similar to semantic information of the first task; and the method further comprises:
when the first success rate is greater than or equal to the preset threshold, determining the first neural network as a neural network used to complete the first task; and
storing the first neural network and the skill library.

### Claim 19 (depends on 12)

The server according to claim 12, wherein the obtaining a first success rate of completing a first task by using a first neural network comprises:
inputting the environment status information into the first neural network, to obtain a skill selected by using the first neural network, wherein the environment status information comprises the status information of the intelligent device and the information about the surrounding environment of the intelligent device in the simulated environment corresponding to the first task;
obtaining a control instruction by using the skill selected by using the first neural network, to control the intelligent device to execute the first task;
repeating the foregoing steps until a quantity of times the first task is completed by using the first neural network reaches a first quantity of times;
counting a quantity of times a result of completing the first task is a success in the first quantity of times, to obtain a second quantity of times; and
calculating the first success rate based on the first quantity of times and the second quantity of times.

### Claim 20 (depends on 12)

The server according to claim 12, wherein the obtaining a first success rate of completing a first task by using a first neural network comprises:
obtaining semantic information of the first task, and obtaining semantic information of a second task, wherein the semantic information of the second task is similar to the semantic information of the first task, and the first neural network is obtained by performing, based on a simulated environment corresponding to the second task, training by using a reinforcement learning algorithm; and
obtaining the first success rate based on a similarity between the semantic information of the first task and the semantic information of the second task, wherein a higher similarity indicates a higher first success rate.