# US 2023/0067970 A1 · Electronic device for providing information for reinforcement learning and method for operating thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0067970 A1
- Document ID: US20230067970A1
- Status: Published application
- Kind: A1
- Application number: 17/891,273
- Filed: 2022-08-19
- Published: 2023-03-02
- Earliest claimed: 2021-08-19
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Hoejoo LEE; Youngcheol JANG
- Web page: https://patentgrep.com/patent/US20230067970A1

## CPC

- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 20/00: Machine learning
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- H04W 24/02: Arrangements for optimising operational condition
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/092: Reinforcement learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

According to various embodiments, a method of operating an electronic device may include: storing at least one value corresponding to each of at least one parameter associated with a radio access network (RAN), and information associated with an operation performed by the RAN. A value corresponding to at least some parameters among the at least one parameter may be used when at least one operation determination model executed by the electronic device determines at least a part of the information associated with the operation. The method of the electronic device may further include: identifying a request for experience information for learning a first operation determination model from a first learning learner for learning the first operation determination model among the at least one operation determination model, identifying, from the at least one value, a value corresponding to at least one first parameter corresponding to the first operation determination model and information associated with an operation corresponding to the at least one first parameter, in response to the request, and providing at least some among the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and a reward value identified based on at least a part of the value corresponding to the at least one first parameter, to the first learning learner as the experience information.

## Claims

### Claim 1

A method of operating an electronic device, the method comprising:
storing at least one value corresponding to each of at least one parameter associated with a radio access network (RAN), and information associated with an operation performed by the RAN, wherein, a value corresponding to at least some parameters among the at least one parameter is used based on at least one operation determination model executed by the electronic device determining at least a part of the information associated with the operation;
identifying a request for experience information for learning a first operation determination model from a first learning learner for learning the first operation determination model among the at least one operation determination model;
in response to the request, identifying, among the at least one value, a value corresponding to at least one first parameter corresponding to the first operation determination model and information associated with an operation corresponding to the at least one first parameter; and
providing at least some among the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and a reward value identified based on at least a part of the value corresponding to the at least one first parameter, to the first learning learner as the experience information.

### Claim 2 (depends on 1)

The method of claim 1, wherein the storing of the at least one value corresponding to each of the at least one parameter associated with the RAN, and the information associated with the operation performed by the RAN comprises:
classifying and storing, for each of a plurality of points in time, the at least one value corresponding to each of the at least one parameter and the information associated with the operation performed by the RAN.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
obtaining, from the first learning learner, a first operation determination model updated based on the provided experience information;
obtaining a new value corresponding to the at least one first parameter from the RAN;
obtaining information associated with a new operation which is a result obtained by applying the new value corresponding to the at least one first parameter to the updated first operation determination model; and
providing the information associated with the new operation to the RAN.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
obtaining, from the first learning learner, the first operation determination model updated based on the provided experience information;
identifying a parameter used by the updated first operation determination model as at least one second parameter which is at least partially different from the at least one first parameter;
obtaining a value corresponding to the at least one second parameter from the RAN;
obtaining information associated with a new operation which is a result obtained by applying the value corresponding to the at least one second parameter to the updated first operation determination model; and
providing the information associated with the new operation to the RAN.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
identifying a new request for experience information for learning the first operation determination model;
in response to the new request, identifying a value corresponding to the at least one second parameter and information associated with an operation corresponding to the at least one second parameter; and
providing, as new experience information, at least some among the value corresponding to the at least one second parameter, the information associated with the operation corresponding to the at least one second parameter, and a reward value identified based on at least a part of the value corresponding to the at least one second parameter.

### Claim 6 (depends on 1)

The method of claim 1, wherein, in response to the request, the identifying of the value corresponding to the at least one first parameter corresponding to the first operation determination model among the at least one value and the information associated with the operation corresponding to the at least one first parameter comprises:
identifying whether at least one value corresponding to each of the at least one parameter supports the at least one first parameter, and based on the at least one first parameter being supported, identifying the value corresponding to the at least one first parameter corresponding to the first operation determination model and the information associated with the operation corresponding to the at least one first parameter.

### Claim 7 (depends on 1)

The method of claim 1, wherein the providing of at least some among the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and the reward value identified based on at least a part of the value corresponding to the at least one first parameter to the first learning learner as the experience information comprises:
providing the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and an entirety of the value corresponding to the at least one first parameter, and an entirety of a reward value identified based on the entirety to the first learning learner as the experience information.

### Claim 8 (depends on 1)

The method of claim 1, wherein the providing of at least some among the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and the reward value identified based on at least a part of the value corresponding to the at least one first parameter to the first learning learner as the experience information comprises:
selecting a part among the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and the value corresponding to the at least one first parameter, and providing the selected part and a reward value identified based on the selected part to the first learning learner as the experience information.

### Claim 9 (depends on 8)

The method of claim 8, wherein the selecting of the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and the part of the value corresponding to the at least one first parameter comprises;
selecting the part among the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and the value corresponding to the at least one first parameter based on at least one operation among: selecting the part based on priority of each of the at least one first parameter, selecting the part based on a point in time at which each value corresponding to the at least one first parameter is obtained, or selecting the part in a random manner

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
identifying the reward value based on a reward determination scheme and at least a part of the value corresponding to the at least one first parameter,
wherein the reward determination scheme is stored in advance in the electronic device or is received by the electronic device.

### Claim 11 (depends on 1)

The method of claim 1, wherein, in response to the request, the identifying of the value corresponding to the at least one first parameter corresponding to the first operation determination model among the at least one value, and the information associated with the operation corresponding to the at least one first parameter comprises;
identifying the at least one first parameter declared by the first operation determination model.

### Claim 12 (depends on 1)

The method of claim 1, wherein, in response to the request, the identifying of the value corresponding to the at least one first parameter corresponding to the first operation determination model among the at least one value, and the information associated with the operation corresponding to the at least one first parameter comprises;
identifying the at least one first parameter based on an external input.

### Claim 13 (depends on 1)

The method of claim 1, wherein the experience information comprises: a value corresponding to the at least one first parameter at a first point in time, a first operation performed by the RAN at the first point in time, a value corresponding to the at least one first parameter at a second point in time after the first point in time according to a result of performing the first operation, and a reward value at the first point in time.

### Claim 14

An electronic device, comprising:
a storage device; and
at least one processor operatively connected to the storage device,
wherein the at least one processor is configured to:
store, in the storage device, at least one value corresponding to each of at least one parameter associated with a radio access network (RAN), and information associated with an operation performed by the RAN, wherein, a value corresponding to at least some parameters among the at least one parameter is used based on at least one operation determination model executed by the electronic device determining at least a part of the information associated with the operation,
identify a request for experience information for learning a first operation determination model from a first learning leaner for learning the first operation determination model among the at least one operation determination model;
in response to the request, identify a value corresponding to at least one first parameter corresponding to the first operation determination model among the at least one value and information associated with an operation corresponding to the at least one first parameter; and
provide at least some among the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and a reward value identified based on at least a part of the value corresponding to the at least one first parameter, to the first learning learner as the experience information.

### Claim 15 (depends on 14)

The electronic device of claim 14, wherein, in response to the request, as at least a part of the identifying of the value corresponding to the at least one first parameter corresponding to the first operation determination model among the at least one value and the information associated with the operation corresponding to the at least one first parameter,
the at least one processor is configured to: identify whether at least one value corresponding to each of the at least one parameter supports the at least one first parameter, and
based on the at least one first parameter being supported, identify the value corresponding to the at least one first parameter corresponding to the first operation determination model and the information associated with the operation corresponding to the at least one first parameter.

### Claim 16 (depends on 14)

The electronic device of claim 14, wherein, as at least a part of the providing of at least some among the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and the reward value identified based on at least a part of the value corresponding to the at least one first parameter to the first learning learner as the experience information,
the at least one processor is configured to: select a part among the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and the value corresponding to the at least one first parameter, and
provide the selected part and a reward value identified based on the selected part to the first learning leaner as the experience information.

### Claim 17

A method of operating an electronic device, the method comprising:
storing a key performance indicator (KPI) table in which at least one value corresponding to each of at least one parameter associated with a radio access network (RAN), and information associated with an operation performed by the RAN are classified for each of a plurality of points in time, wherein, a value corresponding to at least some parameters among the at least one parameter is used when at least one operation determination model executed by the electronic device determines at least a part of the information associated with the operation;
detecting an event for updating a first operation determination model among the at least one operation determination model;
in response to the event detection, identifying a value corresponding to at least one first parameter corresponding to the first operation determination model and information associated with an operation corresponding to the at least one first parameter, by referring to the KPI; and
providing at least some among the value corresponding to the at least one first parameter, the information associated with the operation corresponding to the at least one first parameter, and a reward value identified based on at least a part of the value corresponding to the at least one first parameter, to a first learning leaner for learning the first operation determination model as the experience information.

### Claim 18 (depends on 17)

The method of claim 17, wherein the detecting of the event for updating the first operation determination model comprises;
detecting the event based on a threshold period elapsing from a point in time at which another experience information produced by the electronic device is provided, before the experience information.

### Claim 19 (depends on 17)

The method of claim 17, wherein the detecting of the event for updating the first operation determination model comprises;
detecting the event based on at least a part of the value corresponding to the at least one first parameter satisfying a designated condition.

### Claim 20 (depends on 17)

The method of claim 17, wherein the detecting of the event for updating the first operation determination model comprises;
detecting the event based on a request from the first learning learner and/or an external input.
