# US 12,600,032 B2 · Intermediate policy training system and method for agent > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,600,032 B2 - Document ID: US12600032B2 - Status: Granted - Kind: B2 - Application number: 18/810,481 - Filed: 2024-08-20 - Published: 2025-12-18 - Granted: 2026-04-14 - Earliest claimed: 2024-06-12 - Assignees: Inventec (Pudong) Technology Corporation; Inventec Corporation - Applicants: Inventec (Pudong) Technology Corporation; Inventec Corporation - Inventors: Guilherme Henrique Galelli Christmann; Ying-Sheng Luo; Wei-Chao Chen - Web page: https://patentgrep.com/patent/US12600032B2 ## CPC - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1605: {Simulation of manipulator lay-out, design, modelling of manipulator} - B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor} - G05B 19/4155: characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program - G05B 2219/50391: Robot - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract An intermediate policy training method for agent is provided. This method includes: selecting a source and a target policy from a plurality of policies. Each policy is configured to drive an agent to perform a plurality of actions to be in a plurality of states. Each state includes a plurality of physical properties. The method further includes: respectively selecting one from the source policy and the target policy as a source state and a target state; training an intermediate policy by a reinforcement learning to transition the agent from the source state to the target state over an episode, where the reinforcement learning includes an annealing function for setting a plurality of tolerance boundaries of the plurality of physical properties, and the tolerance boundaries gradually shrink during the episode. ## Claims ### Claim 1 An intermediate policy training method for agent, performed by a processor and comprising: selecting a source policy and a target policy from a plurality of policies, wherein each of the plurality of policies is configured to drive an agent to perform a plurality of actions to be in a plurality of states, with each of the plurality of states comprising a plurality of physical properties; respectively selecting one of the plurality of states from the source policy and the target policy as a source state and a target state; and training an intermediate policy by a reinforcement learning to transition the agent from the source state to the target state over an episode, wherein the reinforcement learning includes an annealing function for setting a plurality of tolerance boundaries of the plurality of physical properties, and the plurality of tolerance boundaries gradually shrink during the episode. ### Claim 2 (depends on 1) The intermediate policy training method for agent of claim 1, wherein a reward function of the reinforcement learning comprises: the intermediate policy receiving a basic reward when the plurality of physical properties remains within the plurality of tolerance boundaries respectively at the end of the episode; the intermediate policy receiving a negative reward when the plurality of physical properties violates any of the plurality of tolerance boundaries before the end of the episode; and the intermediate policy receiving an additional reward greater than the basic reward when the agent is in the target state and the plurality of physical properties remains within the plurality of tolerance boundaries before the end of the episode. ### Claim 3 (depends on 1) The intermediate policy training method for agent of claim 1, wherein the agent is a robot, the robot comprises a plurality of joints and a plurality of proportional-derivative controllers for driving the plurality of joints, and the physical properties comprises: position, orientation, linear velocities, angular velocities, joint angles, and joint velocities. ### Claim 4 (depends on 3) The intermediate policy training method for agent of claim 3, further comprising: deducting a weighted sum of torques of the plurality of joints from a reward obtained by the intermediate policy. ### Claim 5 (depends on 3) The intermediate policy training method for agent of claim 3, wherein the intermediate policy is a feed-forward neural network of two layers, with the two layers having 512 and 256 hidden units respectively, an activation function of the feed-forward neural network is an exponential linear unit, and inputs of the intermediate policy comprises: the target state, the plurality of tolerance boundaries, and a plurality of historical states and a plurality of historical actions of the agent, and an output of the intermediate policy is configured to set the plurality of proportional-derivative controllers. ### Claim 6 An intermediate policy training system for agent, comprising: a storage device configured to store a plurality of instructions; and a processor electrically connected to the storage device to execute the plurality of instructions and cause a plurality of operations, and the plurality of operations comprises: selecting a source policy and a target policy from a plurality of policies, wherein each of the plurality of policies is configured to drive an agent to perform a plurality of actions to be in a plurality of states, with each of the plurality of states comprising a plurality of physical properties; respectively selecting one of the plurality of states from the source policy and the target policy as a source state and a target state; and training an intermediate policy by a reinforcement learning to transition the agent from the source state to the target state over an episode, wherein the reinforcement learning includes an annealing function for setting a plurality of tolerance boundaries of the plurality of physical properties, and the plurality of tolerance boundaries gradually shrink during the episode. ### Claim 7 (depends on 6) The intermediate policy training system for agent of claim 6, wherein a reward function of the reinforcement learning comprises: the intermediate policy receiving a basic reward when the plurality of physical properties remains within the plurality of tolerance boundaries respectively at the end of the episode; the intermediate policy receiving a negative reward when the plurality of physical properties violates any of the plurality of tolerance boundaries before the end of the episode; and the intermediate policy receiving an additional reward greater than the basic reward when the agent is in the target state and the plurality of physical properties remains within the plurality of tolerance boundaries before the end of the episode. ### Claim 8 (depends on 6) The intermediate policy training system for agent of claim 6, wherein the agent is a robot, the robot comprises a plurality of joints and a plurality of proportional-derivative controllers for driving the plurality of joints, and the physical properties comprises: position, orientation, linear velocities, angular velocities, joint angles, and joint velocities. ### Claim 9 (depends on 8) The intermediate policy training system for agent of claim 8, further comprising: deducting a weighted sum of torques of the plurality of joints from a reward obtained by the intermediate policy. ### Claim 10 (depends on 8) The intermediate policy training system for agent of claim 8, wherein the intermediate policy is a feed-forward neural network of two layers, with the two layers having 512 and 256 hidden units respectively, an activation function of the feed-forward neural network is an exponential linear unit, and inputs of the intermediate policy comprises: the target state, the plurality of tolerance boundaries, and a plurality of historical states and a plurality of historical actions of the agent, and an output of the intermediate policy is configured to set the plurality of proportional-derivative controllers.