# US 2025/0094769 A1 · Hybrid agent for parameter optimization using prediction and reinforcement learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094769 A1
- Document ID: US20250094769A1
- Status: Published application
- Kind: A1
- Application number: 18/843,217
- Filed: 2023-02-16
- Published: 2025-03-20
- Earliest claimed: 2022-03-15
- Assignees: Telefonaktiebolaget LM Ericsson (publ)
- Applicants: Telefonaktiebolaget LM Ericsson (publ)
- Inventors: Jaeseong JEONG; Wenfeng HU; Konstantinos VANDIKAS; Alexandros NIKOU; Maxim TESLENKO
- Web page: https://patentgrep.com/patent/US20250094769A1

## CPC

- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/092: Reinforcement learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method for reinforcement learning includes obtaining a first observation ot-1 of an environment at an end of a first time step t-1, generating a prediction o′t of a second observation ot of the environment at the end of a second time step t based on at least the first observation, obtaining a predicted state s′t of the environment at the second time step t from the predicted second observation o′t, selecting an action at to execute on the environment during the second time step based on the predicted state s′t and a policy π, and executing the action at on the environment.

## Claims

### Claim 1

A computer-implemented method, comprising:
obtaining a first observation ot-1 of an environment at an end of a first time step t-1;
generating a prediction o′t of a second observation ot of the environment at the end of a second time step t based on at least the first observation;
obtaining a predicted state s′t of the environment at the second time step t from the predicted second observation o′t;
selecting an action at to execute on the environment during the second time step based on the predicted state s′t and a policy π; and
executing the action at on the environment.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
obtaining the second observation ot of the environment following execution of the action at;
determining a reward rt based on the second observation ot; and
updating the policy π based on the reward rt.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating the prediction o′t of the second observation ot is performed by applying a sequence prediction model to a plurality of previous observations of the environment to obtain the prediction o′t of the second observation ot.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
determining an accuracy of the prediction; and
adjusting a number of the previous observations used by the sequence prediction model in response to the determined accuracy.

### Claim 5 (depends on 4)

The method of claim 4, wherein adjusting the number of the previous observations used by the sequence prediction model in response to the determined accuracy comprises increasing the number of the previous observations used by the sequence prediction model in response to determining that the accuracy of the prediction is less than a threshold level of accuracy.

### Claim 6 (depends on 3)

The method of claim 3, wherein the sequence prediction model comprises a multivariate time series forecasting model.

### Claim 7 (depends on 6)

The method of claim 6, wherein the multivariate time series forecasting model comprises a long short term memory, LSTM, model.

### Claim 8 (depends on 5)

The method of claim 5, wherein generating the prediction o′t of the second observation ot is performed based on observations from previous time steps, and is performed based on an assumption that a predetermined action a0 is taken at the second time step.

### Claim 9 (depends on 8)

The method of claim 8, wherein the predetermined action comprises no action.

### Claim 10 (depends on 8)

The method of claim 8, wherein generating the prediction o′t of the second observation of is performed based on side channel information about the environment in addition to the observations from previous time steps.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
determining that an action taken in the first time step was the predetermined action a0; and;
training a prediction model, that is used to generate the prediction o′t of the second observation ot, based on an actual observation ot-1 at time step t-1.

### Claim 12 (depends on 1)

The method of claim 1, wherein the first observation ot-1 comprises a static state component, an action dependent state component and a confounder state component, wherein the action dependent state component is a variable component that is dependent on actions taken on the environment and the confounder state is a variable component that is substantially independent of actions taken on the environment.

### Claim 13 (depends on 12)

The method of claim 12, wherein the confounder state has a predictable time-series pattern.

### Claim 14 (depends on 1)

The method of claim 1, wherein the environment comprises a computer-controlled system, and wherein the first and second observations comprise observations of a performance indicator of the system.

### Claim 15 (depends on 14)

The method of claim 14, wherein the action comprises a modification of a configurable parameter of the system, the parameter impacting the performance indicator.

### Claim 16 (depends on 14)

The method of claim 14, wherein the environment comprises a wireless communication network, and wherein the observation comprises one or more key performance indicators, KPIs, of the wireless communication network.

### Claim 17 (depends on 16)

The method of claim 16, wherein the KPIs comprise a reference signal received power, an interference level, an average downlink signal to interference plus noise ratio, SINR, an average uplink SINR, a nominal uplink power, average data rate, throughput and/or an average uplink neighbor SINR.

### Claim 18 (depends on 16)

The method of claim 16, wherein the action comprises a modification of a configurable network parameter of the wireless communication network.

### Claim 19 (depends on 18)

The method of claim 18, wherein the configurable network parameter comprises a downlink transmit power, an uplink transmit power, and/or an antenna tilt.

### Claim 20 (depends on 1)

A control system, comprising:
a processor;
a communication interface coupled to the processor; and
a memory coupled to the processor, wherein the memory comprises computer readable instructions that when executed by the processor cause the system to perform operations according to claim 1.

### Claim at position 21

21-22. (canceled)