# US 2021/0341886 A1 · System and Method of Efficient, Continuous, and Safe Learning Using First Principles and Constraints

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0341886 A1
- Document ID: US20210341886A1
- Status: Published application
- Kind: A1
- Application number: 17/319,442
- Filed: 2021-05-13
- Published: 2021-11-04
- Earliest claimed: 2018-11-16
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Lifeng Liu; Yingxuan Zhu; Jun Zhang; Xiaofian Yin; Jian Li; Yongxiang Tao; Dayao Liang
- Web page: https://patentgrep.com/patent/US20210341886A1

## CPC

- G05B 13/0265: {the criterion being a learning criterion}
- G06N 20/00: Machine learning
- B60W 50/06: Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
- G06F 16/2457: with adaptation to user needs
- G06F 16/24578: {using ranking}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- B60W 60/00: Drive control systems specially adapted for autonomous road vehicles
- B60W 60/001: {Planning or execution of driving tasks}
- B60W 2050/0062: {Adapting control system settings}
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer implemented method for self-learning of a control system. The method includes creating an initial knowledge base. The method learns first principles using the knowledge base. The method creates initial control commands derived from the knowledge base. The method generates constraints for the control commands. The method performs constrained reinforcement learning by executing the control commands with the constraints and observing feedback to improve the control commands. The method enriches the knowledge base based on the feedback.

## Claims

### Claim 1

A computer implemented method for self-learning of a control system, the method comprising:
creating a knowledge base comprising data obtained from real-world experiments;
learning first principles using the knowledge base, wherein the first principles are foundational principles that cannot be deduced from other principles;
creating control commands based on the first principles derived from the knowledge base;
generating constraints for the control commands;
performing constrained reinforcement learning by executing the control commands with the constraints,
utilizing feedback from the constrained reinforcement learning to improve the control commands; and
enriching the knowledge base based on the feedback.

### Claim 2 (depends on 1)

The method of claim 1, wherein creating control commands based on the first principles derived from the knowledge base comprises:
calculating dynamics and kinematics parameters; and
generating the control commands based on the dynamics and kinematics parameters.

### Claim 3 (depends on 1)

The method of claim 1, wherein creating the control commands derived from the knowledge base comprises receiving a current state, a target state, and a reference to the knowledge base as input parameters.

### Claim 4 (depends on 3)

The method of claim 3, wherein creating the control commands derived from the knowledge base comprises:
breaking a control task down into separate components according to the current state and the target state;
creating a query for each of the separate components of the control task;
retrieving a corresponding query result from the knowledge base for each query to generate a plurality of query results; and
combining the query results to generate a control command for the control task.

### Claim 5 (depends on 4)

The method of claim 4, wherein the query results are combined according to corresponding weights assigned to the separate components of the control task.

### Claim 6 (depends on 1)

The method of claim 1, wherein the constraints for the control commands include hard constraints that specify conditions that cannot be exceeded.

### Claim 7 (depends on 1)

The method of claim 1, wherein the constraints for the control commands include soft constraints that specify preferable conditions.

### Claim 8 (depends on 1)

The method of claim 1, wherein generating constraints for the control commands comprises:
generating a first subset of constraint items based on a state of an operating environment;
generating a second subset of constraint items for a set of filtered moving objects based on a state of a host machine and a target state;
generating a third subset of constraint items for a set of filtered stationary obstacles; and
combining the first subset of constraint items, the second subset of constraint items, and the third subset of constraint items.

### Claim 9 (depends on 1)

The method of claim 1, wherein performing the constrained reinforcement learning to improve the control commands comprises:
decomposing the control commands into multiple categories and dimensions to enable learning of the control commands in each category and dimension separately;
generating control command candidates for a control command based on a current state;
applying the constraints to the control commands candidates;
refining the control commands based on past experiences when the current state is a learned state; and
adapting results from learned states to new environments when the current state is not the learned state.

### Claim 10 (depends on 1)

The method of claim 1, wherein enriching the knowledge base based on the feedback comprises refining a manifestation of dynamics and kinematics models of the knowledge base.

### Claim 11

A system comprising
a memory storage unit comprising instructions; and
one or more processors in communication with the memory storage unit, wherein the one or more processors execute the instructions to:
create a knowledge base comprising data obtained from real-world experiments;
learn first principles using the knowledge base, wherein the first principles are foundational principles that cannot be deduce from other principles;
create control commands based on the first principles derived from the knowledge base;
generate constraints for the control commands;
perform constrained reinforcement learning by executing the control commands with the constraints,
utilize feedback from the constrained reinforcement learning to improve the control commands; and
enrich the knowledge base based on the feedback.

### Claim 12 (depends on 11)

The system of claim 11, wherein the one or more processors further execute the instructions to:
calculate dynamics and kinematics parameters; and
generate the control commands based on the dynamics and kinematics parameters.

### Claim 13 (depends on 11)

The system of claim 11, wherein the one or more processors further execute the instructions to receive a current state, a target state, and a reference to the knowledge base as input parameters.

### Claim 14 (depends on 13)

The system of claim 13, wherein the one or more processors further execute the instructions to:
break a control task down into separate components according to the current state and the target state;
create a query for each of the separate components of the control task;
retrieve a corresponding query result from the knowledge base for each query to generate a plurality of query results; and
combine the query results to generate a control command for the control task.

### Claim 15 (depends on 14)

The system of claim 14, wherein the one or more processors further execute the instructions to combine the query results according to corresponding weights assigned to the separate components of the control task.

### Claim 16 (depends on 11)

The system of claim 11, wherein the constraints for the control commands include hard constraints that specify conditions that cannot be exceeded.

### Claim 17 (depends on 11)

The system of claim 11, wherein the constraints for the control commands include soft constraints that specify preferable conditions.

### Claim 18 (depends on 11)

The system of claim 11, wherein the one or more processors further execute the instructions to:
generate a first subset of constraint items based on a state of an operating environment;
generate a second subset of constraint items for a set of filtered moving objects based on a state of a host machine and a target state;
generate a third subset of constraint items for a set of filtered stationary obstacles; and
combine the first subset of constraint items, the second subset of constraint items, and the third subset of constraint items.

### Claim 19 (depends on 11)

The system of claim 11, wherein the one or more processors further execute the instructions to:
decompose the control commands into multiple categories and dimensions to enable learning of the control commands in each category and dimension separately;
generate control command candidates for a control command based on a current state;
apply the constraints to the control commands candidates;
refine the control commands based on past experiences when the current state is a learned state; and
adapt results from learned states to new environments when the current state is not the learned state.

### Claim 20

A non-transitory computer readable medium storing computer instructions, that when executed by one or more processors, cause the one or more processors to perform the steps of:
creating a knowledge base comprising data obtained from real-world experiments;
learning first principles using the knowledge base, wherein the first principles are foundational principles that cannot be deduce from other principles;
creating control commands based on the first principles derived from the knowledge base;
generating constraints for the control commands;
performing constrained reinforcement learning by executing the control commands with the constraints,
utilizing feedback from the constrained reinforcement learning to improve the control commands; and
enriching the knowledge base based on the feedback.
