# US 11,027,430 B2 · Systems and methods for latency compensation in robotic teleoperation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,027,430 B2
- Document ID: US11027430B2
- Status: Granted
- Kind: B2
- Application number: 16/159,218
- Filed: 2018-10-12
- Published: 2020-04-16
- Granted: 2021-06-08
- Earliest claimed: 2018-10-12
- Assignees: Toyota Research Institute, Inc.
- Applicants: Toyota Research Institute, Inc.
- Inventors: Allison Thackston; Sam Zapolsky; Katarina Bouma; Laura Stelzner; Ron Goldman
- Web page: https://patentgrep.com/patent/US11027430B2

## CPC

- B25J 9/1689: {Teleoperation}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J 9/1697: {Vision controlled systems}
- G05B 2219/40151: Time delay, problems caused by time delay between local and remote
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A method includes presenting a virtual representation of an environment of a robot, receiving a first user command to control the robot within the environment, rendering a predicted version of the virtual representation during a period of latency in which current data pertaining to the environment of the robot is not available, updating the predicted version of the virtual representation based upon a second user command received during the period of latency, and upon conclusion of the period of latency, reconciling the predicted version of the virtual representation with current data pertaining to the environment of the robot.

## Claims

### Claim 1

A method comprising:
presenting a virtual representation of an environment of a robot;
receiving a first user command to control the robot within the environment;
rendering a predicted version of the virtual representation during a period of latency in which current data pertaining to the environment of the robot is not available;
updating the predicted version of the virtual representation based upon a second user command received during the period of latency, wherein the second user command relates to directing the robot to manipulate an object in the environment of the robot;
receiving, during the period of latency, a third user command to complete interaction of the robot with the object in the environment;
simulating completion of the interaction of the robot with the object; and
upon conclusion of the period of latency, reconciling (i) the predicted version of the virtual representation with current data pertaining to the environment of the robot and (ii) the predicted version of the object now disengaged from the robot with current data about the object.

### Claim 2 (depends on 1)

The method of claim 1 further comprising providing haptic feedback based upon a predicted state of the robot within the predicted version of the virtual representation during the period of latency for which the current data is not available.

### Claim 3 (depends on 1)

The method of claim 1 further comprising providing audio feedback based upon a predicted state of the robot within the predicted version of the virtual representation during the period of latency for which the current data is not available.

### Claim 4 (depends on 1)

The method of claim 1 wherein the second user command to control the robot is not related to the motility of the robot and differs from the first user command.

### Claim 5 (depends on 1)

The method of claim 1 wherein the predicted version of the virtual representation is rendered during the period of latency utilizing preexisting data pertaining to the environment of the robot.

### Claim 6 (depends on 1)

The method of claim 1 wherein the period of latency is a first period of latency having a duration that differs from a second duration of a second period of latency.

### Claim 7

A method comprising:
presenting a virtual representation of an environment of a robot;
receiving a first user command to control the robot within the environment;
rendering a first predicted version of the virtual representation during a first period of latency in which current data pertaining to the environment of the robot is not available, wherein the first period of latency has a duration that differs from a second duration of a second period of latency, wherein the first predicted version of the virtual representation is reconciled with the current data at the expiration of the first duration;
rendering a second predicted version of the virtual representation during the second period of latency in which current data pertaining to the environment of the robot is not available;
updating the first predicted version of the virtual representation based upon a second user command received during the first period of latency;
upon conclusion of the first period of latency, reconciling the first predicted version of the virtual representation with current data pertaining to the environment of the robot; and
subsequent to the reconciliation of the first predicted version at the expiration of the first duration, reconciling the second predicted version of the virtual representation with second received data corresponding to expiration of the first duration.

### Claim 8 (depends on 1)

The method of claim 1 further comprising reconciling the predicted version of the virtual representation based upon the current data and data corresponding to the latency period pertaining to the environment of the robot.

### Claim 9

An interface device comprising:
memory; and
a processor coupled to the memory, the processor being configured to:
present a virtual representation of an environment of a robot;
receive a first user command to control the robot within the environment;
render a predicted version of the virtual representation during a period of latency in which current data pertaining to the environment of the robot is not available;
update the predicted version of the virtual representation based upon a second user command received during the period of latency;
receive, during the period of latency, a third user command to complete interaction of the robot with the object in the environment;
simulate completion of the interaction of the robot with the object; and
upon conclusion of the period of latency, reconcile (i) the predicted version of the virtual representation with current data pertaining to the environment of the robot and (ii) the predicted version of the object now disengaged from the robot with current data about the object.

### Claim 10 (depends on 9)

The system of claim 9 wherein the processor is further configured to provide haptic feedback based upon a predicted state of the robot within the predicted version of the virtual representation during the period of latency for which the current data is not available.

### Claim 11 (depends on 9)

The system of claim 9 wherein the processor is further configured to provide audio feedback based upon a predicted state of the robot within the predicted version of the virtual representation during the period of latency for which the current data is not available.

### Claim 12 (depends on 9)

The system of claim 9 wherein the second user command relates to directing the robot to manipulate an object in the environment of the robot.

### Claim 13 (depends on 9)

The system of claim 9 wherein the second user command to control the robot is not related to the motility of the robot and differs from the first user command.

### Claim 14 (depends on 9)

The system of claim 9 wherein the processor is further configured to render the predicted version of the virtual representation during the period of latency utilizing preexisting data pertaining to the environment of the robot.

### Claim 15 (depends on 9)

The system of claim 9 wherein the period of latency is a first period of latency having a duration that differs from a second duration of a second period of latency.

### Claim 16 (depends on 9)

The system of claim 9 wherein the predicted version of the virtual representation is a first predicted version of the virtual representation reconciled with the current data at the expiration of the first duration, the system being further configured to:
render a second predicted version of the virtual representation during the second period of latency in which current data pertaining to the environment of the robot is not available;
subsequent to the reconciliation of the first predicted version at the expiration of the first duration, reconcile a second predicted version of the virtual representation with second received data corresponding to expiration of the first duration.

### Claim 17 (depends on 9)

The system of claim 9 wherein the processor is further configured to reconcile the predicted version of the virtual representation based upon the current data and data corresponding to the latency period pertaining to the environment of the robot.