# US 10,572,775 B2 · Learning and applying empirical knowledge of environments by robots

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,572,775 B2
- Document ID: US10572775B2
- Status: Granted
- Kind: B2
- Application number: 15/832,705
- Filed: 2017-12-05
- Published: 2019-06-06
- Granted: 2020-02-25
- Earliest claimed: 2017-12-05
- Assignees: X Development LLC
- Applicants: X Development LLC
- Inventors: Alexa Greenberg
- Web page: https://patentgrep.com/patent/US10572775B2

## CPC

- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1697: {Vision controlled systems}
- G05B 13/027: {using neural networks only}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G05B 2219/33038: Real time online learning, training, dynamic network
- G05B 2219/39046: Compare image of plate on robot with reference, move till coincidence, camera
- G05B 2219/39543: Recognize object and plan hand shapes in grasping movements
- G06T 2207/20081: Training; Learning
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques described herein relate to generating a posteriori knowledge about where objects are typically located within environments to improve object location. In various implementations, output from vision sensor(s) of a robot may include visual frame(s) that capture at least a portion of an environment in which a robot operates/will operate. The visual frame(s) may be applied as input across a machine learning model to generate output that identifies potential location(s) of an object of interest. The robot's position/pose may be altered based on the output to relocate one or more of the vision sensors. One or more subsequent visual frames that capture at least a not-previously-captured portion of the environment may be applied as input across the machine learning model to generate subsequent output identifying the object of interest. The robot may perform task(s) that relate to the object of interest.

## Claims

### Claim 1

A method implemented by one or more processors, comprising:
determining an object of interest;
receiving vision data, the vision data generated based on vision sensor output from one or more vision sensors of a vision component of a robot, the vision data including one or more visual frames that capture at least a portion of an environment in which a robot operates or will operate;
applying one or more of the visual frames as input across a machine learning model to generate output, wherein the output identifies one or more probabilities that a plurality of pixels representing one or more surfaces in the portion of the environment captured in the one or more visual frames that conceal, from a vantage point of the one or more vision sensors, an instance of the object of interest; and
altering a position or pose of the robot based on the output to relocate one or more of the vision sensors to have a direct view behind one or more of the surfaces;
wherein the machine learning model was trained using one or more annotated vision frames in which a portion of a surface of the one or more annotated vision frames is annotated as concealing another object of interest of a same type as the object of interest.

### Claim 2 (depends on 1)

The method of claim 1, wherein the machine learning model comprises a convolutional neural network.

### Claim 3 (depends on 1)

The method of claim 1, wherein the input applied across the machine learning model includes a reduced dimensionality embedding of the object of interest.

### Claim 4

A method implemented by one or more processors, comprising:
determining an object of interest;
receiving vision data, the vision data generated based on vision sensor output from one or more vision sensors of a vision component of a robot, the vision data including at least one visual frame that captures at least a portion of an environment in which a robot operates or will operate;
applying the at least one visual frame as input across a machine learning model to generate output, wherein the output identifies one or more other portions of the environment that are outside of the portion of the environment captured by the at least one visual frame, wherein the one or more other portions of the environment potentially include an instance of the object of interest;
altering a position or pose of the robot based on the output to relocate one or more of the vision sensors to have a direct view of a given other portion of the one or more other portions of the environment;
obtaining, from one or more of the vision sensors, at least one subsequent visual frame that captures the given other portion of the environment;
applying the at least one subsequent visual frame as input across the machine learning model to generate subsequent output, wherein the subsequent output identifies the instance of the object of interest; and
operating the robot to perform one or more tasks that relate to the instance of the object of interest.

### Claim 5 (depends on 4)

The method of claim 4, wherein the machine learning model comprises a convolutional neural network.

### Claim 6 (depends on 4)

The method of claim 4, wherein the input applied across the machine learning model includes a reduced dimensionality embedding of the object of interest.