# US 12,110,867 B2 · Wind turbine control based on reinforcement learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,110,867 B2
- Document ID: US12110867B2
- Status: Granted
- Kind: B2
- Application number: 17/639,925
- Filed: 2020-08-13
- Published: 2022-10-13
- Granted: 2024-10-08
- Earliest claimed: 2019-09-16
- Assignees: Siemens Gamesa Renewable Energy A/S
- Applicants: Siemens Gamesa Renewable Energy A/S
- Inventors: Bert Gollnick
- Web page: https://patentgrep.com/patent/US12110867B2

## CPC

- F03D 7/026: {for starting-up}
- F03D 7/046: {with learning or adaptive control, e.g. self-tuning, fuzzy logic or neural network}
- G05B 13/027: {using neural networks only}
- F05B 2270/404: active, predictive, or anticipative
- F05B 2270/709: with neural networks
- Y02E 10/72: Wind turbines with rotation axis in wind direction
- F03D: WIND MOTORS
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- F05B: INDEXING SCHEME RELATING TO WIND, SPRING, WEIGHT, INERTIA OR LIKE MOTORS, TO MACHINES OR ENGINES FOR LIQUIDS COVERED BY SUBCLASSES F03B, F03D AND F03G
- Y02E: REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION

## Abstract

Methods, systems, and devices for wind turbine control based on reinforcement learning are disclosed. The method comprises receiving data indicative of a current environmental state of the wind turbine, determining one or more controlling actions of the wind turbine based on the current environmental state of the wind turbine and a reinforcement learning algorithm, and applying the determined one or more controlling actions to the wind turbine.

## Claims

### Claim 1

A method of controlling a wind turbine, comprising:
receiving data indicative of a current environmental state of the wind turbine, wherein the current environmental state of the wind turbine specifies a plurality of the following: a wind speed, a turbulence intensity, a site elevation, a temperature, a vertical wind shear, or a horizontal wind shear;
determining one or more controlling actions of the wind turbine based on the current environmental state of the wind turbine and a reinforcement learning algorithm, wherein the one or more controlling actions comprise at least one of keep-idling or start-up;
predicting a reward value for a future point in time for possible chains of controlling actions selected from a predefined set of controlling actions, respectively, by processing the current environmental state together with a chain of controlling actions using a predefined value function of the reinforcement learning algorithm, wherein the one or more chains of controlling actions are determined based on a highest predicted reward value for the future point in time for the chain of controlling actions; and
controlling the wind turbine by applying the one or more controlling actions to the wind turbine.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more controlling actions are determined based on a predefined set of controlling actions, wherein each of the controlling actions in the predefined set of controlling actions is associated with a reference reward value, and wherein each of the one or more controlling actions is determined at the current environmental state.

### Claim 3 (depends on 2)

The method of claim 2, wherein the one or more controlling actions are determined based on the predefined set of controlling actions taking the respective reference reward value into account.

### Claim 4 (depends on 2)

The method of claim 2, wherein the one or more controlling actions are determined based on the predefined set of controlling actions using an arbitrary selection.

### Claim 5 (depends on 2)

The method of claim 2, further comprising:
upon applying the one or more controlling actions: receiving measurement data associated with a current reward value;
performing a comparison between the reference reward value and the current reward value; and
training the reinforcement learning algorithm based on the comparison.

### Claim 6 (depends on 2)

The method of claim 2, wherein the reward value comprises a net power output of the wind turbine.

### Claim 7 (depends on 1)

The method of claim 1, wherein the value function comprises a linear function or a neural network.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
upon applying the one or more controlling actions: receiving measurement data associated with a current aggregate reward value;
performing a comparison between the predicted reward value and the current aggregate reward value; and
training the reinforcement learning algorithm based on the comparison.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
pre-training the reinforcement learning algorithm by using tuples of environmental states, controlling actions, and net power outputs which are obtained from other wind turbines or expert knowledge.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
performing a prediction of future environmental states using an environmental states prediction model, wherein the environmental states prediction model receives as input a current weather information and information on a mechanical structure of the wind turbine to generate future environmental states.

### Claim 11

A device for controlling a wind turbine, the device comprising one or more processors, the one or more processors being configured to:
receive data indicative of a current environmental state of the wind turbine, wherein the current environmental state of the wind turbine specifies a plurality of the following: a wind speed, a turbulence intensity, a site elevation, a temperature, a vertical wind shear, or a horizontal wind shear;
determine one or more controlling actions of the wind turbine based on the current environmental state of the wind turbine and a reinforcement learning algorithm, wherein the one or more controlling actions comprise at least one of keep-idling or start-up;
predict a reward value for a future point in time for possible chains of controlling actions selected from a predefined set of controlling actions, respectively, by processing the current environmental state together with a chain of controlling actions using a predefined value function of the reinforcement learning algorithm, wherein the one or more chains of controlling actions are determined based on a highest predicted reward value for the future point in time for the chain of controlling actions; and
control the wind turbine by applying one or more controlling actions to the wind turbine.

### Claim 12 (depends on 11)

The device of claim 11, wherein the one or more processors are further configured to:
perform a prediction of future environmental states using an environmental states prediction model, wherein the environmental states prediction model receives as input a current weather information and information on a mechanical structure of the wind turbine to generate future environmental states.