# US 12,111,884 B2 · Optimal sequential decision making with changing action space

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,111,884 B2
- Document ID: US12111884B2
- Status: Granted
- Kind: B2
- Application number: 17/659,983
- Filed: 2022-04-20
- Published: 2023-10-26
- Granted: 2024-10-08
- Earliest claimed: 2022-04-20
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Tanay Anand; Pinkesh Badjatiya; Sriyash Poddar; Jayakumar Subramanian; Georgios Theocharous; Balaji Krishnamurthy
- Web page: https://patentgrep.com/patent/US12111884B2

## CPC

- G06F 18/2137: based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/092: Reinforcement learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for machine learning are described. Embodiments of the present disclosure receive state information that describes a state of a decision making agent in an environment; compute an action vector from an action embedding space based on the state information using a policy neural network of the decision making agent, wherein the policy neural network is trained using reinforcement learning based on a topology loss that constrains changes in a mapping between an action set and the action embedding space; and perform an action that modifies the state of the decision making agent in the environment based on the action vector, wherein the action is selected based on the mapping.

## Claims

### Claim 1

A method of machine learning, comprising:
receiving, by a monitoring component, state information that describes a state of a decision making agent in an environment;
computing, using a policy neural network of the decision making agent, an action vector from an action embedding space based on the state information, wherein the policy neural network is trained using reinforcement learning based on a topology loss that constrains changes in a mapping between an action set and the action embedding space; and
performing, by the decision making agent, an action that modifies the state of the decision making agent in the environment based on the action vector, wherein the action is selected based on the mapping.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
computing, by the policy neural network, an action embedding of the action based on the mapping;
calculating, by an action component, a distance between the action embedding and the action vector in the action embedding space; and
selecting, by the action component, the action based on the distance between the action embedding and the action vector.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the topology loss is based on a level of exploration in the environment.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
computing, by a learning component, a reward based on the modified state; and
updating, by the learning component, parameters of the policy neural network based on the reward.

### Claim 5 (depends on 4)

The method of claim 4, wherein:
the parameters of the policy neural network include parameters corresponding to the mapping.

### Claim 6 (depends on 5)

The method of claim 5, wherein:
the parameters corresponding to the mapping are updated based on the topology loss.

### Claim 7 (depends on 4)

The method of claim 4, further comprising:
embedding, by the learning component, the reward in the action embedding space, wherein the parameters of the policy neural network are updated based on the embedding of the reward.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
monitoring, by the monitoring component, user interactions between a user and a set of content items, wherein the state information is based on the user interactions; and
recommending, by an action component, a content item from the set of content items to the user, wherein the action includes the recommendation.

### Claim 9

A method for machine learning, comprising:
receiving, by a monitoring component, state information of a decision making agent;
selecting, by an action component, an action from an action set based on the state information using a policy neural network of the decision making agent;
computing, by a learning component, a reward based on the action;
computing, by the learning component, a topology loss that minimizes changes in a mapping between the action set and an action embedding space; and
updating, by the learning component, parameters of the policy neural network based on the reward and the topology loss.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
adding, by the action component, a new action to the action set; and
updating, by the action component, the mapping to include the new action.

### Claim 11 (depends on 10)

The method of claim 10, wherein:
the mapping is updated based on the topology loss.

### Claim 12 (depends on 9)

The method of claim 9, further comprising:
computing, by the action component, distances between each of a plurality of action embeddings in the action embedding space, wherein each of the plurality of action embeddings corresponds to an action from the action set, and wherein the topology loss is based on the distances.

### Claim 13 (depends on 9)

The method of claim 9, further comprising:
determining, by the learning component, parameters of the mapping based on competitive Hebbian learning.

### Claim 14 (depends on 9)

The method of claim 9, further comprising:
computing, by the learning component, a decay value, wherein the topology loss is based on the decay value.

### Claim 15 (depends on 9)

The method of claim 9, further comprising:
embedding, by the learning component, the reward in the action embedding space, wherein the parameters of the policy neural network are updated based on the embedding of the reward.

### Claim 16

An apparatus for machine learning, comprising:
a monitoring component configured to receive state information indicating a state of an environment;
a policy neural network configured to compute an action vector from an action embedding space based on the state information, wherein the policy neural network is trained using reinforcement learning based on a topology loss that constrains changes in a mapping between an action set and the action embedding space; and
an action component configured to perform an action that modifies the state based on the action vector, wherein the action is selected based on the mapping.

### Claim 17 (depends on 16)

The apparatus of claim 16, further comprising:
a learning component configured to compute a reward based on the action, and to update parameters of the policy neural network based on the reward.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein:
the learning component is configured to compute distances between each of a plurality of action embeddings in the action embedding space, wherein each of the plurality of action embeddings corresponds to an action from the action set, and wherein the topology loss is based on the distances.

### Claim 19 (depends on 16)

The apparatus of claim 16, further comprising:
a learning component configured to compute a decay value, and wherein the topology loss is based on the decay value.

### Claim 20 (depends on 16)

The apparatus of claim 16, further comprising:
a learning component configured to embed a reward in the action embedding space, and wherein parameters of the policy neural network are updated based on the embedding of the reward.