# US 2026/0017530 A1 · Invertible-Reasoning Policy and Reverse Dynamics for Causal Reinforcement Learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0017530 A1
- Document ID: US20260017530A1
- Status: Published application
- Kind: A1
- Application number: 19/327,623
- Filed: 2025-09-12
- Published: 2026-01-15
- Earliest claimed: 2023-03-17
- Applicants: CCNets, Inc.
- Inventors: Jun Ho PARK
- Web page: https://patentgrep.com/patent/US20260017530A1

## CPC

- G06N 3/092: Reinforcement learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed herein is the framework for Causal Reinforcement Learning which combines Causal Cooperative Networks with the Actor-Critic algorithm, introducing reverse dynamics, and invertible-reasoning policy within the framework to enable bidirectional transitions while maximizing accumulative rewards. The framework involves a redesigned Critic and Actor, as well as a newly developed Reverse-environment network. During the iterative training and exploration phases, the cooperative network learns a policy that identifies actions capable of reversing a future state back to its prior state, through the reverse-environment network. It allows agents to consider the consequences of their actions, which facilitates deeper decision-making and exploration strategies.

## Claims

### Claim 1

A method for reinforcement learning of neural networks, comprising:
a cooperative network configured to receive a sample of transition data, consisting of a state, action, reward, and the next state of an agent within an environment, and to learn a policy and value function that control a transition from a future state to its prior state, implemented through neural networks of a Critic, an Actor, and a Reverse-environment, wherein:
the Critic estimates a value of a sampled state and transmits the estimated value to the Actor and Reverse-environment;
the Actor infers an action, from the sampled state and the estimated value, and transmits the inferred action to the Reverse-environment; and
the Reverse-environment outputs a state that is recurred, from the sampled next state based on the inferred action and the estimated value, and outputs a state that is reversed from the sampled next state based on a sampled action and the estimated value,
wherein costs associated with the forward, reverse, and recurrent transitions, between the sampled state and the next state, are calculated based on a discrepancy among the sampled state, the reversed state, and the recurred state, and
wherein an error within the cooperative network is derived from a combination of the forward, reverse, and recurrent costs.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the transition data is obtained through an exploration of the agent within the environment, and
the actor determines an action for the agent, based on the current state and a value estimate received from the critic.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the forward cost is calculated based on a discrepancy between the recurred state to the reversed state;
the reverse cost is calculated based on a discrepancy between the reversed state to the sampled state; and
the recurrent cost is calculated based on a discrepancy between the recurred state to the sampled state.

### Claim 4 (depends on 3)

The method of claim 3, wherein:
the forward cost is associated with a critic error and/or an actor error in the cooperative network;
the reverse cost is associated with a critic error and/or a reverse-environment error in the cooperative network; and
the recurrent cost is associated with an actor error and/or a reverse-environment error in the cooperative network.

### Claim 5 (depends on 4)

The method of claim 4, wherein:
the cooperative critic error is derived from a difference between the sum of the forward and reverse costs, and the recurrent cost; and
the cooperative actor error is derived from a difference between the sum of the recurrent and forward costs, and the reverse cost; and
the cooperative reverse-environment error is derived from a difference between the sum of the reverse and recurrent costs, and the forward cost.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
the value difference is calculated as the difference between the expected sum of rewards and the estimated value.

### Claim 7 (depends on 6)

The method of claim 6, wherein:
the expected sum of rewards is computed based on a value estimate of the next state and the sampled reward.

### Claim 8 (depends on 7)

The method of claim 7, wherein:
the value estimate of the next state is determined by a target network of the critic network and the target network is a slower-updating copy of the critic network.

### Claim 9 (depends on 6)

The method of claim 6, wherein:
the value function error is derived by minimizing the value difference.

### Claim 10 (depends on 9)

The method of claim 9, wherein:
mean Squared Error (MSE) is employed to minimize the value difference, which is the difference between the expected sum of rewards and the estimated value.

### Claim 11 (depends on 5)

The method of claim 5, wherein:
the critic loss is calculated as the sum of the value function error and the cooperative critic error.

### Claim 12 (depends on 11)

The method of claim 11, wherein:
the backpropagation of the critic loss computes gradients of the loss function with respect to the parameters of the critic network without being involved in adjusting the actor or the reverse-environment; and the parameters of the critic network are adjusted based on the calculated gradients.

### Claim 13 (depends on 5)

The method of claim 5, wherein:
the backpropagation of the cooperative critic error and the value function error computes, respectively, gradients of the error functions with respect to the parameters of the critic network without being involved in adjusting the actor or the reverse-environment; and the parameters of the critic network are adjusted based on the calculated gradients.

### Claim 14 (depends on 5)

The method of claim 5, wherein:
the value difference is referred to as the advantage, and the actor loss is calculated by multiplying the advantage with the cooperative actor error.

### Claim 15 (depends on 14)

The method of claim 14, wherein:
the backpropagation of the actor loss computes gradients of the loss function with respect to the parameters of the actor network without being involved in adjusting the reverse-environment; and the parameters of the actor network are adjusted based on the calculated gradients.

### Claim 16 (depends on 5)

The method of claim 5, wherein:
the backpropagation of the cooperative reverse-environment error computes gradients of the error function with respect to the parameters of the reverse-environment; and
the parameters of the reverse-environment network are adjusted based on the calculated gradients.

### Claim 17

A method for reinforcement learning of neural networks, comprising:
a cooperative network configured to receive a sample of transition data, consisting of a state, action, reward, and the next state of an agent within an environment, and to learn a policy and value function that control a transition from a future state to its prior state, implemented through neural networks of a Critic, an Actor, and a Reverse-environment, wherein:
the Critic estimates a value of a sampled state and transmits the estimated value to the Actor and Reverse-environment;
the Reverse-environment generates a state that is reversed from the sampled next state based on a sampled action and the estimated value;
the Actor outputs an action that is recurred from the reversed state and the estimated value, and outputs an action that is inferred from the sampled state and the estimated value;
wherein costs associated with the forward, reverse, and recurrent transitions, between the sampled state and the next state, are calculated based on a discrepancy among the sampled action, the inferred action, and the recurred action, and
wherein an error within the cooperative network is derived from combination of the forward, reverse, and recurrent costs.

### Claim 18

A method for reinforcement learning of neural networks, comprising:
a cooperative network configured to receive a sample of transition data, consisting of a state, action, reward, and the next state of an agent within an environment, and to learn a policy and value function that control a transition from a future state to its prior state, implemented through neural networks of a Critic, an Actor, and a Reverse-environment, wherein:
the Critic estimates a value vector of a sampled state and transmits the data distribution of the value vector to the Actor and Reverse-environment;
the Actor infers an action from the sampled state and the data distribution and transmits the inferred action to the Reverse-environment; and
the Reverse-environment outputs a state that is recurred from the sampled next state based on the inferred action and the data distribution, and outputs a state that is reversed from the sampled next state based on a sampled action and the data distribution,
wherein costs associated with the forward, reverse, and recurrent transitions, between the sampled state and the next state, are calculated based on a discrepancy among the sampled state, the reversed state, and the recurred state, and
wherein an error within the cooperative network is derived from combination of the forward, reverse, and recurrent costs.
