# US 10,245,724 B2 · System, method and product for utilizing prediction models of an environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,245,724 B2
- Document ID: US10245724B2
- Status: Granted
- Kind: B2
- Application number: 15/177,411
- Filed: 2016-06-09
- Published: 2017-12-14
- Granted: 2019-04-02
- Earliest claimed: 2016-06-09
- Assignees: Shmuel Ur Innovation LTD.
- Applicants: Shmuel Ur Innovation Ltd.
- Inventors: Shmuel Ur; Vlad Dabija; David Hirshberg
- Web page: https://patentgrep.com/patent/US10245724B2

## CPC

- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- B25J 9/1612: {characterised by the hand, wrist, grip control}
- B25J 9/1669: {characterised by special application, e.g. multi-arm co-operation, assembly, grasping}
- B25J 9/1697: {Vision controlled systems}
- G06T 7/12: Edge-based segmentation
- G06T 7/75: {involving models}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G05B 2219/39536: Planning of hand motion, grasping
- G05B 2219/40425: Sensing, vision based motion planning
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A first method comprising: predicting a scene of an environment using a model of the environment and based on a first scene of the environment obtained from sensors observing scenes of the environment; comparing the predicted scene with an observed scene from the sensors; and performing an action based on differences determined between the predicted scene and the observed scene. A second method comprising applying a vibration stimuli on an object via a computer-controlled component; obtaining a plurality of images depicting the object from a same viewpoint, captured during the application of the vibration stimuli. The second method further comprising comparing the plurality of images to detect changes occurring in response to the application of the vibration stimuli, which changes are attributed to a change of a location of a boundary of the object; and determining the boundary of the object based on the comparison.

## Claims

### Claim 1

A method comprising:
obtaining a first scene of an environment from one or more sensors, wherein the one or more sensors observe scenes of the environment;
predicting, using a model of the environment and based on the first scene of the environment, a predicted second scene of the environment;
obtaining an observed second scene from the one or more sensors, wherein the observed second scene occurs later in time than the first scene;
wherein prior to said obtaining the observed second scene, performing computations based on the predicted second scene;
comparing the predicted second scene with the observed second scene; and
performing an action, using the computations and based on differences determined in said comparing between the predicted second scene and the observed second scene, whereby utilizing the computations based on the predicted second scene to reduce response time of performing the action in response to the observed second scene.

### Claim 2 (depends on 1)

The method of claim 1,
wherein said performing the action comprises utilizing the computations, whereby processing time occurring after said obtaining the observed second scene is reduced by relying on the predicted second scene.

### Claim 3 (depends on 1)

The method of claim 1, wherein the computations are computations of an initial movement plan of a robotic device, wherein the initial movement plan is based on the predicted second scene, wherein said performing comprises instructing the robotic device to move in accordance with the initial movement plan or a modification thereof, whereby the robotic device is moved.

### Claim 4 (depends on 2)

The method of claim 2, wherein the computations are computations of a movement plan of a robotic device, wherein the movement plan is based on the predicted second scene, wherein said performing comprises: in response to determining that the observed second scene is identical to the predicted second scene, performing the movement plan, whereby moving the robotic device.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
wherein said performing the action comprises:
compressing the observed second scene based on the differences between the predicted second scene and the observed second scene, wherein said compressing creates a compressed representation of the observed second scene,
whereby the observed second scene is obtainable from the compressed representation of the observed second scene by a device capable of performing a prediction, using the model of the environment, based on the first scene;
whereby the first scene and the observed second scene are retainable in a computer-readable medium using a reduced amount of space in comparison to retaining the first scene and the observed second scene directly.

### Claim 6 (depends on 5)

The method of claim 5, further comprises:
transmitting the first scene of the environment to a computing device;
wherein said performing the action further comprises:
transmitting the compressed representation to the computing device;
whereby the computing device is capable of obtaining the observed second scene by performing a prediction based on the first scene and using the model of the environment to obtain a predicted scene of the environment and modifying the predicted scene using the compressed representation to obtain the observed second scenes,
whereby reducing amount of bandwidth required for transmission the first scene and the second observed scene without losing information.

### Claim 7 (depends on 1)

The method of claim 1, further comprises:
wherein the one or more sensors comprise a first sensor and a second sensor, wherein the first sensor is different than the second sensor;
wherein said obtaining the first scene comprises obtaining the first scene from a first sensor;
wherein said obtaining the observed second scene comprises obtaining the observed scene from a second sensor.

### Claim 8 (depends on 7)

The method of claim 7, wherein the observed second scene occurs later in time than the first scene.

### Claim 9 (depends on 7)

The method of claim 7, wherein the observed second scene occurs at the same time as the first scene.

### Claim 10 (depends on 7)

The method of claim 7, wherein scenes of the environment observed by the first sensor and scenes of the environment observed by the second sensor demonstrate different views of a same location in the environment.

### Claim 11 (depends on 7)

The method of claim 7, wherein the first sensor and the second sensor observe different locations of the environment.

### Claim 12 (depends on 1)

The method of claim 1, wherein the one or more sensors comprise one or more cameras, wherein scenes obtained from the one or more sensors are images of the environment.

### Claim 13

A method comprising:
obtaining a first scene of an environment from one or more sensors, wherein the one or more sensors observe scenes of the environment, wherein the environment comprises a pile of physical objects;
determining a stimuli force to be applied on a physical object of the pile of physical objects;
predicting, using a model of the pile of physical objects and based on the first scene of the environment, a predicted second scene of the environment
in response to application of the stimuli force on the physical object;
wherein after said predicting, applying the stimuli force on the physical object; and
obtaining an observed second scene from the one or more sensors after said applying;
comparing the predicted second scene with the observed second scene; and
performing an action based on differences determined in said comparing between the predicted second scene and the observed second scene, wherein said performing the action comprises updating, based on the differences between the predicted second scene and the observed second scene, the model of the pile.

### Claim 14 (depends on 13)

The method of claim 13, wherein said performing the action further comprises: based on the updated model, determining an order of picking up objects from the pile.

### Claim 15 (depends on 13)

The method of claim 13 further comprises determining, based on said comparing, a mechanical coupling between two or more objects of the pile of physical objects that was not modeled in the model of the pile, wherein said updating the model comprises introducing to the model a modeling of the mechanical coupling.

### Claim 16 (depends on 15)

The method of claim 15 further comprises selecting the object upon which the stimuli force is to be applied, wherein said selecting comprises selecting the object based on a determination that applying stimuli force on the object is expected to provide information useful for validating or refining the model of the pile.

### Claim 17 (depends on 13)

The method of claim 13, wherein the stimuli force is applied by one or more robotic devices.

### Claim 18 (depends on 13)

The method of claim 13, wherein the stimuli force is a vibration force.

### Claim 19 (depends on 18)

The method of claim 18, wherein a frequency of the vibration force is below a sampling frequency of the one or more sensors.

### Claim 20 (depends on 18)

The method of claim 18, wherein a frequency of the vibration force is above a sampling frequency of the one or more sensors, wherein the frequency is correlated with the sampling frequency.

### Claim 21 (depends on 13)

The method of claim 13, wherein an intensity of the stimuli force is below a maximal threshold, wherein the maximal threshold is an intensity capable of permanently changing a structure of the pile.

### Claim 22 (depends on 13)

The method of claim 13, wherein an intensity of the stimuli force is above a minimal threshold, wherein the minimal threshold is an intensity capable of moving the physical object in a manner detectable by the one or more sensors.