US 12,393,175 B2Granted

Systems and methods for skill learning with multiple critics

AssigneeNaver Corporation

InventorsDavid Emukpere, Bingbing Wu, Julien Perez

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems and methods are disclosed for determining a policy to recommend transition in a position-representing space for a robotic device using a multi-critic architecture. To learn policy in a multi-critic architecture, a set of critics is defined pertaining to a position-representing space where each critic corresponds to a different objective function such as reach-reward, discovery-reward, and safety-reward. For each one of the critics of the set of critics, a learned value function in position-representing space is determined. The policy is learned based on the weighted feedback of the learned value functions to recommend transitions that are safe in the position-representing space. The multi-critic architecture minimizes interference between multiple reward functions and learns a safe and stable policy for the robotic device.

Claims20

Dependent claims

A computer-implemented method, comprising:

identifying a set of critics that pertain to a position-representing space, wherein each of the set of critics corresponds to a different objective function, and wherein the set of critics includes a first critic, a second critic, and a third critic;
accessing an objective function of the first critic that is based on a reach-reward objective configured to be anti-correlated with an extent of a movement from an initial position to a target position within the position-representing space;
accessing an objective function of the second critic that is based on a discovery-component objective configured to be correlated with an extent to which the movement triggers expansion of a bound or volume of the position-representing space;
and
accessing an objective function of the third critic that is based on a safety-component objective that is configured to be anticorrelated with an extent to which any of one or more safety constraints are violated during or at a completion of the movement;
identifying, for each critic of the set of critics, a learned value function in the position-representing space that is based on the objective function of the critic;
assigning, to each critic of the set of critics, a weight;
receiving sensor data;
identifying a position within the position-representing space that corresponds to the received sensor data;
updating a policy based on the learned value function for each critic of the set of critics and the weights of the set of critics, wherein the policy corresponds to a movement objective in the position-representing space;
determining, based on the policy, a recommended transition within the position-representing space from the initial position to the target position;
and
outputting a representation of the recommended transition within the position-representing space.

10 more elements

A system comprising:

one or more data processors;
and
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including:
identifying a set of critics that pertain to a position-representing space, wherein each of the set of critics corresponds to a different objective function, and wherein the set of critics includes a first critic, a second critic, and a third critic;
accessing an objective function of the first critic that is based on a reach-reward objective configured to be anti-correlated with an extent of a movement from an initial position to a target position within the position-representing space;
accessing an objective function of the second critic that is based on a discovery-component objective configured to be correlated with an extent to which the movement triggers expansion of a bound or volume of the position-representing space;
and
accessing an objective function of the third critic that is based on a safety-component objective that is configured to be anticorrelated with an extent to which any of one or more safety constraints are violated during or at a completion of the movement;
identifying, for each critic of the set of critics, a learned value function in the position-representing space that is based on the objective function of the critic;
assigning, to each critic of the set of critics, a weight;
receiving sensor data;
identifying a position within the position-representing space that corresponds to the received sensor data;
updating a policy based on the learned value function for each critic of the set of critics and the weights of the set of critics, wherein the policy corresponds to a movement objective in the position-representing space;
determining, based on the policy, a recommended transition within the position-representing space from the initial position to the target position;
and
outputting a representation of the recommended transition within the position-representing space.

13 more elements

A computer-implemented method for learning a policy to recommend transitions in a position-representing space for a robotic device, comprising:

accessing, with one or more processors, a set of reward functions of a multi-critic architecture having a set of critics, wherein each critic of the set of critics is associated with a reward function of the set of reward functions that is one of:
(i) a safety reward function;
(ii) an exploration reward function;
and (iii) an accuracy reward function;
and
learning, with one or more processors, the policy based on learned value functions associated with the set of critics;
wherein said learning includes iteratively performing a set of operations until convergence of a set of parameters of the policy is detected, wherein the set of operations includes:
sampling a sequence of skills for one or more rollouts of the policy, wherein the accuracy reward function assesses performance of the sequence of skills;
collecting one or more trajectories, wherein each of the one or more trajectories is a trajectory of part or all of a robotic device that has or is to have its movement evaluated using the policy or is a trajectory of an object moved by the robotic device;
updating, based on the one or more trajectories, a state representation space that is received as input to the learned value functions associated with the set of critics and that represents:
(i) potential positions of part or all of the robotic device and/or of the object moved by the robotic device;
and/or (ii) potential velocities of part or all of the robotic device and/or of the object moved by the robotic device;
updating the learned value functions associated with the set of critics using the updated state representation space;
computing, for each of the set of critics, an advantage of a given potential action relative to other potential actions using an advantage function associated with the critic;
generating an advantage score by normalizing and combining the advantages of the set of critics;
and
updating the set of policy parameters based on the advantage score.

14 more elements

A computer-implemented method for using a policy to recommend transitions in a position-representing space for a robotic device, the computer-implemented method comprising:

receiving, with one or more processors, a representation of a task for the robotic device;
computing, using the policy, a set of recommended transitions in the position-representing space for performing the task;
and
triggering a movement of the robotic device by initiating the set of recommended transitions;
wherein the policy is trained by:
accessing, with one or more processors, a set of reward functions of a multi-critic architecture having a set of critics, wherein each critic of the set of critics is associated with a reward function of the set of reward functions that is one of:
(i) a safety reward function;
(ii) an exploration reward function;
and (iii) an accuracy reward function;
and
learning, with one or more processors, the policy based on learned value functions associated with the set of critics;
wherein said learning includes iteratively performing a set of operations until convergence of a set of parameters of the policy is detected, wherein the set of operations includes:
sampling a sequence of skills for one or more rollouts of the policy, wherein the accuracy reward function assesses performance of the sequence of skills;
collecting one or more trajectories, wherein each of the one or more trajectories is a trajectory of part or all of a robotic device that has or is to have its movement evaluated using the policy or is a trajectory of an object moved by the robotic device;
updating, based on the one or more trajectories, a state representation space that is received as input to the learned value functions associated with the set of critics and that represents:
(i) potential positions of part or all of the robotic device and/or of the object moved by the robotic device;
and/or (ii) potential velocities of part or all of the robotic device and/or of the object moved by the robotic device;
updating the learned value functions associated with the set of critics using the updated state representation space;
computing, for each of the set of critics, an advantage of a given potential action relative to other potential actions using an advantage function associated with the critic;
generating an advantage score by normalizing and combining the advantages of the set of critics;
and
updating the set of policy parameters based on the advantage score.

19 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/511,829Filed 16 Nov 2023
Published: US 2025/0164966 A122 May 2025
Granted: 19 Aug 2025
Term ends: Not estimatedThe source records don’t support an estimate.

Classification5 codes

G05B 19/4155
Numerical control [NC], i.e. automatically operating machines, in particular machine tools, e.g. in a manufacturing environment, so as to…
B25J 9/163
Characterised by the control loop: learning, adaptive, model based, rule based expert control
G05B 2219/39376
Robotics, robotics to robotics hand: hierarchical, learning, recognition and skill level and adaptation servo level
G05B
Control or regulating systems in general