# US 11,992,944 B2 · Data-efficient hierarchical reinforcement learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,992,944 B2 - Document ID: US11992944B2 - Status: Granted - Kind: B2 - Application number: 17/050,546 - Filed: 2019-05-17 - Published: 2021-06-24 - Granted: 2024-05-28 - Earliest claimed: 2018-05-18 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Honglak Lee; Shixiang Gu; Sergey Levine - Web page: https://patentgrep.com/patent/US11992944B2 ## CPC - B25J 9/1602: {characterised by the control system, structure, architecture} - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1656: {characterised by programming, planning systems for manipulators} - G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour - G06N 3/0499: Feedforward networks - G06N 3/092: Reinforcement learning - G06N 20/00: Machine learning - G05B 2219/39289: Adaptive ann controller - G06N 3/08: Learning methods - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Training and/or utilizing a hierarchical reinforcement learning (HRL) model for robotic control. The HRL model can include at least a higher-level policy model and a lower-level policy model. Some implementations relate to technique(s) that enable more efficient off-policy training to be utilized in training of the higher-level policy model and/or the lower-level policy model. Some of those implementations utilize off-policy correction, which re-labels higher-level actions of experience data, generated in the past utilizing a previously trained version of the HRL model, with modified higher-level actions. The modified higher-level actions are then utilized to off-policy train the higher-level policy model. This can enable effective off-policy training despite the lower-level policy model being a different version at training time (relative to the version when the experience data was collected). ## Claims ### Claim 1 A method of off-policy training of a higher-level policy model of a hierarchical reinforcement learning model for use in robotic control, the method implemented by one or more processors and comprising: retrieving, from previously stored experience data, for a robot, generated based on controlling the robot during a previous experience episode using the hierarchical reinforcement learning model in a previously trained state: a stored state based on an observed state of the robot in the previous experience episode; a stored higher-level action for transitioning from the stored state to a goal state; wherein the stored higher-level action was previously generated, during the previous experience episode, using the higher-level policy model, and wherein the stored higher-level action was previously processed, during the previous episode using a lower-level policy model of the hierarchical reinforcement learning model, in generating a lower-level action applied to the robot during the previous experience episode; and at least one stored environment reward determined based on application of the lower-level action during the previous episode; determining a modified higher-level action to utilize in lieu of the stored higher-level action for further training of the hierarchical reinforcement learning model, wherein determining the modified higher-level action is based on a currently trained state of the lower-level policy model, the currently trained state of the lower-level policy model differing from the previously trained state; and further off-policy training the higher-level policy model using the stored state, using the at least one stored environment reward, and using the modified higher-level action in lieu of the stored higher-level action. ### Claim 2 (depends on 1) The method of claim 1, further comprising: subsequent to the further off-policy training, using the hierarchical reinforcement learning model to control one or more actuators of an additional robot. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the modified higher-level action based on the stored higher-level action and based on the currently trained state of the lower-level policy model, comprises: identifying a plurality of candidate modified higher-level actions, including the modified higher-level action; and selecting the modified higher-level action based on determining that the modified higher-level action has a greater probability, than other of the candidate modified higher-level actions, of inducing the lower-level action if applied with the stored state observation to the lower-level policy model in the currently trained state of the lower-level policy model. ### Claim 4 (depends on 3) The method of claim 3, further comprising: selecting the modified higher-level action over the stored higher-level action based on determining that the modified higher-level action has a greater probability, than the stored higher-level action, of inducing the lower-level action if applied with the stored state to the lower-level policy model in the currently trained state of the lower-level policy model. ### Claim 5 (depends on 3) The method of claim 3, wherein identifying the plurality of candidate modified higher-level actions comprises selecting a fixed number of candidate modified higher-level actions. ### Claim 6 (depends on 5) The method of claim 5, wherein selecting the fixed number of candidate modified higher-level actions is based on a random sampling from a Gaussian that is centered based on the stored state observation. ### Claim 7 (depends on 1) The method of claim 1, wherein the stored state includes positions for a plurality of components of the robot. ### Claim 8 (depends on 7) The method of claim 7, wherein the stored state lacks any velocity for any components of the robot. ### Claim 9 (depends on 1) The method of claim 1, further comprising: off-policy training the lower-level policy model based on a parameterized reward determined based on: the higher-level action, and a given state of the robot, during the previous experience episode, that resulted from application of the lower-level action. ### Claim 10 (depends on 9) The method of claim 9, further comprising: determining the parameterized reward based on distance between the goal state and the given state. ### Claim 11 (depends on 1) The method of claim 1, wherein the previous experience episode is performed in simulation and the robot is a simulated robot. ### Claim 12 (depends on 1) The method of claim 1, wherein the environment reward is determined based on a state observation following application of the lower-level action during the previous episode, and wherein the environment reward is determined based on a task for which the higher-level policy model is being trained. ### Claim 13 A method implemented by one or more processors, the method comprising: identifying a current state observation of a robot; determining, using a higher-level policy model of a hierarchical reinforcement learning model, a higher-level action for transitioning from the current state observation to a goal state observation; generating an atomic action based on processing the current state observation and the higher-level action using a lower-level policy model of the hierarchical reinforcement learning model; applying the atomic action to the robot to cause the robot to transition to an updated state; generating an intrinsic reward for the atomic action, the intrinsic reward generated based on the updated state and the goal state observation; and training the lower-level policy model based on the intrinsic reward for the atomic action. ### Claim 14 (depends on 13) The method of claim 13, further comprising: subsequent to the training, using the hierarchical reinforcement learning model to control one or more actuators of an additional robot. ### Claim 15 (depends on 13) The method of claim 13, wherein the robot is a simulated robot. ### Claim 16 (depends on 13) The method of claim 13, wherein the generating the intrinsic reward based on the updated state and the goal state observation comprises generating the intrinsic reward based on an L2 difference between the updated state and the goal state observation. ### Claim 17 (depends on 13) The method of claim 13, further comprising generating an environment reward, and training the higher-level policy model based on the environment reward. ### Claim 18 A method implemented by one or more processors of a robot, the method comprising: identifying a current state of the robot; determining, at a first control step and using a higher-level policy model of a hierarchical reinforcement learning model, a higher-level action for transitioning from the current state to a goal state; generating a first lower-level action for the first control step based on processing the current state and the higher-level action using a lower-level policy model of the hierarchical reinforcement learning model; applying the first lower-level action to the robot to cause the robot to transition to an updated state; generating, at a second control step that follows the first control step, an updated higher-level action, wherein generating the updated higher-level action comprises applying at least the current state, the updated state, and the higher-level action to a transition function; generating a second lower-level action for the second control step based on processing the updated state and the updated higher-level action using the lower-level policy model; and applying the second lower-level action to the robot to cause the robot to transition to a further updated state. ### Claim 19 (depends on 18) The method of claim 18, further comprising: generating a first intrinsic reward for the first lower-level action, the first intrinsic reward generated based on the updated state and the goal state; generating a second intrinsic reward for the second lower-level action, the second intrinsic reward generated based on the further updated state and the goal state; and training the lower-level policy model based on the first and second intrinsic rewards. ### Claim 20 (depends on 19) The method of claim 19, wherein: generating the first intrinsic reward based on the updated state and the goal state comprises generating the first intrinsic reward based on an L2 difference between the updated state and the goal state observation; and generating the second intrinsic reward based on the further updated state and the goal state comprises generating the second intrinsic reward based on an L2 difference between the further updated state and the goal state observation.