# US 2021/0341895 A1 · Pid controller autotuner using machine learning approaches

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0341895 A1
- Document ID: US20210341895A1
- Status: Published application
- Kind: A1
- Application number: 16/861,863
- Filed: 2020-04-29
- Published: 2021-11-04
- Earliest claimed: 2020-04-29
- Assignees: Honeywell International Inc.
- Applicants: Honeywell International Inc.
- Inventors: Vladimir Havlena; Daniel Pachner
- Web page: https://patentgrep.com/patent/US20210341895A1

## CPC

- G05B 19/0425: {Safety, monitoring}
- G05B 11/42: for obtaining a characteristic which is both proportional and time-dependent, e.g. P. I., P. I. D.
- G05B 13/0265: {the criterion being a learning criterion}
- G05B 19/0426: {Programming the control sequence}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G05B 2219/13095: Pid regulator
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An approach for auto tuning a PID controller that may incorporate determining set-points for controlled variables, obtaining controlled variables from a controlled process module, taking a difference between the set-points and the controlled variable as inputs to a proportional, integral and derivative (PID) controller, calculating a loss or reward from the difference between the controlled variables and the set-points, and achieving set-point tracking. The loss or reward needs to be an increasing or decreasing function of a control error value. Also incorporated may be adding loss or reward components based on controlled variables time difference or control action time difference, which may effect a self-tuned performance of the PID controller.

## Claims

### Claim 1

A PID controller system comprising:
a proportional, integral and derivative (PID) controller;
a machine learning module having an output connection to the PID controller; and
a controlled process module that provides controlled variables to the PID controller via a differentiator that also takes in set-points for the controlled variables and the differentiator outputs a control error to the PID controller, wherein the control error is a difference between the set-points and the controlled variables; and
wherein:
the controlled variables and set-points for the controlled variables are input to the machine learning module; and
the machine learning module calculates an instantaneous loss or reward which is an increasing or decreasing, respectively, function of the control error.

### Claim 2 (depends on 1)

The system of claim 1, wherein additional loss or reward components are based on a controlled variables absolute time difference or control action absolute time difference.

### Claim 3 (depends on 1)

The system of claim 1, wherein the machine learning module comprises:
a reinforcement learning module that provides the output from the machine learning module to the PID controller;
a loss or reward calculator having inputs of controlled variables, set-points for controlled variables, and control action, and having an output to the reinforcement learning module, a differentiator having an input of a control action from the PID controller and having an output to the reinforcement learning module; and
a process states augmentation and differentiation module having inputs of process states, controlled variables, and set-points for controlled variables, and having an output to the reinforcement learning module.

### Claim 4 (depends on 3)

The system of claim 3, wherein an output of the reinforcement learning module provides auto tuning for the PID controller that includes integral action that ensures a steady state offset free tracking of set-points by relevant process variables.

### Claim 5 (depends on 4)

The system of claim 4, wherein using process state and control time difference and augmenting differentiated states with control errors before use as data for the reinforcement learning module, are effected.

### Claim 6 (depends on 5)

The system of claim 5, wherein a process state is defined as controlled process variables values and time differences thus producing a standard PID linear controller having proportional, integral and derivative actions for each process variable including optimized loop interactions.

### Claim 7 (depends on 5)

The system of claim 5, wherein the auto tuning mechanism has a multi-objective loss function consisting of multiple losses with a pre-specified non-negative weighting factor where a Q-function has weighting factors as parameters and produces an optimal control law for any values of the weighting factors at any time without a restart of an auto tuner needed.

### Claim 8 (depends on 7)

The system of claim 7, wherein sufficient statistics relevant for an action state value function estimation being collected in an edge controller and transferring the statistics along to a cloud computer that provides reinforcement learning results in sending an updated control law back to the edge controller.

### Claim 9

A method for auto tuning a PID controller comprising:
determining set-points for controlled variables;
obtaining controlled variables from a controlled process module;
taking a difference between the set-points and the controlled variable as inputs to a proportional, integral and derivative (PID) controller;
calculating a loss or reward from the difference between the controlled variables and the set-points; and
achieving set-point tracking, wherein the loss or reward needs to be an increasing or decreasing function of a control error value.

### Claim 10 (depends on 9)

The method of claim 9, further comprising adding loss or reward components based on a controlled variables time difference or control action time difference.

### Claim 11 (depends on 10)

The method of claim 10, wherein the adding the loss or reward components effect a self-tuned performance of the PID controller.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
passing an output of a control action to a controlled process module; and
having an output of process states from the controlled process module proceed as an input to a machine learning module; and
wherein process states contain information that the machine learning module uses to develop with a reinforcement learning algorithm to produce control laws that converge to an optimal control law that is sent to the PID controller to optimize performance of the PID controller.

### Claim 13 (depends on 12)

The method of claim 12, wherein:
reinforcement learning produces integral action when it is configured to optimize a controlled process input time difference; and
control action supplied to reinforced learning is a time difference of the control action applied to the controlled process module.

### Claim 14

An auto tuner system comprising:
a machine learning module;
a proportional, integral and derivative (PID) controller having an input connected to the machine learning module;
a controlled process module having an input connected to the PID controller; and
a differentiator having an input connected to the controlled process module and an output connected to the PID controller.

### Claim 15 (depends on 14)

The system of claim 14, wherein the machine learning module comprises a loss or reward value calculator having an input of set-points for controlled variables, a second input of controlled variables from the controlled process module, a third input of a control action from the PID controller, and an output to a reinforcement learning submodule.

### Claim 16 (depends on 15)

The system of claim 15, wherein the machine learning module further comprises a process states augmentation and differentiation submodule having a first input of controlled variables from the controlled process module, a second input of set-points for the controlled variables, a third input of process states from the controlled process module, and an output to the reinforcement learning module.

### Claim 17 (depends on 16)

The system of claim 16, wherein the machine learning module further comprises a differentiation submodule having an input of control action from the PID controller and an output to the reinforcement learning module.

### Claim 18 (depends on 17)

The system of claim 17, wherein the input of the differentiator receives control variables from the controlled process module, a second input of the differentiator receives set-points for the controlled variables, and an output provides a control error to the PID controller.

### Claim 19 (depends on 18)

The system of claim 18, wherein the control error is integrated at the PID controller.

### Claim 20 (depends on 14)

The system of claim 14, wherein the PID controller and the machine learning module are deployed on the same microcontroller hardware.
