# US 10,482,658 B2 · Visualization and control of remote objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,658 B2
- Document ID: US10482658B2
- Status: Granted
- Kind: B2
- Application number: 14/675,488
- Filed: 2015-03-31
- Published: 2015-10-01
- Granted: 2019-11-19
- Earliest claimed: 2014-03-31
- Applicants: Gary Stephen Shuster; Charles Marion Curry
- Inventors: Gary Stephen Shuster; Charles Marion Curry
- Web page: https://patentgrep.com/patent/US10482658B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems, devices and methods for controlling remote devices by modification of visual data prior to presentation to a person in order to make the person's response effectively the same as if the person were responding to data transmitted, processed and acted on instantaneously are disclosed. The systems, devices and methods advantageously minimize or eliminate the risks caused by a human response to data that has been delayed in transmission, processing and presentation. In an embodiment, a person controlling a remote device using an augmented reality interface is able to control the device based on predicted positions of an object at the time action is taken, thereby advantageously compensating for delays in receiving data, acting on such data and transmitting instructions or a response to the remote device.

## Claims

### Claim 1

A method for control of a remote device, the method comprising:
analyzing by a computing device operably coupled to the remote device, a data stream to identify a speed and a trajectory of a moving object separate from the remove device, wherein the data stream is analyzed by using digital signal analysis and object recognition algorithms;
determining by the computing device, at least one predicted path based, at least in part, on the trajectory and the speed of the moving object, wherein the at least one predicted path is selected from two or more possible paths and the two or more possible paths are based on physical characteristics of the object and an initial path on which the object is travelling;
altering the data stream to display the moving object on the at least one predicted path;
displaying the moving object on the at least one predicted path to at least one human operator;
receiving control input from the at least one human operator;
comparing the at least one predicted path to an actual path;
determining the at least one predicted path matches the actual path;
in response to determining that the at least one predicted path matches the actual path, controlling the remote device based on the control input by the at least one human operator.

### Claim 2 (depends on 1)

The method of claim 1, further comprising calculating a probability that the moving object will be on the at least one predicted path.

### Claim 3 (depends on 1)

The method of claim 1, further comprising, if the probability of two or more predicted paths is greater than a threshold probability, displaying the two or more predicted paths to a first human operator.

### Claim 4 (depends on 1)

The method of claim 1, further comprising, if the probability of two or more predicted paths is greater than a threshold probability, displaying different predicted paths to different human operators.

### Claim 5 (depends on 4)

The method of claim 4, wherein an opacity of the different predicted paths displayed relates to the likelihood that the object will be on a given predicted path.

### Claim 6 (depends on 1)

The method of claim 1, further comprising, calculating an amount of time between generation of the data stream and perception of the data stream by the at least one human operator, and altering the amount of time to compensate for a flash-drag effect of the at least one human operator.

### Claim 7 (depends on 1)

The method of claim 1, wherein the trajectory or the speed are determined based on an instantaneous trajectory, an instantaneous speed, a changing trajectory, a changing speed, or a combination thereof.

### Claim 8 (depends on 1)

The method of claim 1, further comprising sending one or more control signals, instructions, or responses from the at least one human operator to the remote device when the actual path and one of the at least one predicted path are the same within a predetermined tolerance.

### Claim 9 (depends on 8)

The method of claim 8, further comprising adjusting the at least one predicted path to compensate for a transmission time for the control signal, instruction, or response from the at least one human operator to be received by the remote device.

### Claim 10 (depends on 1)

The method of claim 1, further comprising negating or altering a control signal, instruction, or response from the at least one human operator when the actual path and one of the at least one predicted path is less than a predetermined tolerance.

### Claim 11

A system for compensating for the delay in controlling a remote device, the system comprising:
a computing device operably coupled to the remote device, the computing device configured to:
analyze, using digital signal analysis and object recognition algorithms, a data stream to identify a speed and a trajectory of a moving object separate from the remove device;
determine a predicted path of the moving object based, at least in part on the trajectory and the speed of the moving object, wherein the predicted path is selected from two or more possible paths;
alter the data stream to show the moving object on the predicted path;
display the moving object on the predicted path to a human operator;
receive control input from the human operator indicating a response;
if an actual path is different from the predicted path, analyze elements in the predicted path and the actual path;
if the elements are more than a threshold percentage similar, recalculate the control input from the human operator to deliver a same response to the actual path as the response of the human operator to the predicted path.

### Claim 12 (depends on 11)

The system of claim 11, wherein the predicted path is adjusted to compensate for encryption and decryption times of the data stream.

### Claim 13 (depends on 11)

The system of claim 11, wherein the predicted path is adjusted to compensate for processing time of the human operator and nerve signal to limb movement conversion time.

### Claim 14 (depends on 13)

The system of claim 13, wherein the predicted path is further adjusted to compensate for a transmission time of the control input from the human operator to the remote device.

### Claim 15 (depends on 11)

The system of claim 11, wherein the computing device is further configured to determine a predicted response of the human operator.

### Claim 16 (depends on 15)

The system of claim 15, wherein the computing device is further configured to calculate a confidence level of the predicted response of the human, and when the confidence level is greater than a threshold level, instruct the remote device to implement the predicted response.