# US 2021/0081791 A1 · Computer-Automated Robot Grasp Depth Estimation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081791 A1
- Document ID: US20210081791A1
- Status: Published application
- Kind: A1
- Application number: 17/020,565
- Filed: 2020-09-14
- Published: 2021-03-18
- Earliest claimed: 2019-09-13
- Assignees: Osaro
- Applicants: Osaro
- Inventors: Ben Goodrich; Alex Kuefler; William D. Richards; Christopher Correa; Rishi Sharma; Sulabh Kumra
- Web page: https://patentgrep.com/patent/US20210081791A1

## CPC

- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- G06N 3/08: Learning methods
- G05B 19/4155: characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program
- G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06T 7/73: using feature-based methods
- G05B 2219/39271: Ann artificial neural network, ffw-nn, feedforward neural network
- G05B 2219/39484: Locate, reach and grasp, visual guided grasping
- G05B 2219/40269: Naturally compliant robot arm
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06T 2200/04: involving 3D image data
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A computer system trains a neural network to predict, for each pixel in an input image, the position that a robot's end effector would reach if a grasp (“poke”) were attempted at that position. Training data consists of images and end effector positions recorded while a robot attempts grasps in a pick-and-place environment. For an automated grasping policy, the approach is self-supervised, as end effector position labels may be recovered through forward kinematics, without human annotation. Although gathering such physical interaction data is expensive, it is necessary for training and routine operation of state of the art manipulation systems. Therefore, the system comes “for free” while collecting data for other tasks (e.g., grasping, pushing, placing). The system achieves significantly lower root mean squared error than traditional structured light sensors and other self-supervised deep learning methods on difficult, industry-scale jumbled bin datasets.

## Claims

### Claim 1

A method performed by at least one computer processor executing computer program instructions stored on at least one non-transitory computer-readable medium, the method comprising:
(A) training a neural network to predict, for each of a plurality of pixels P in an input image, a depth that a robot's end effector would reach in a three-dimensional environment if the robot's end effector moved into the three-dimensional environment to target a projection of the pixel P onto the three-dimensional environment, the method comprising:
(A)(1) receiving, as training data:
a plurality of training images I, each representing the three-dimensional scene at a corresponding time when the robot's end effector is at a corresponding stopping point in the three-dimensional scene; and
a plurality of poses of the robot's end effector, each representing a corresponding pose of the robot's end effector at a corresponding one of the stopping points; and
(A)(2) training the neural network using the plurality of training images I, and the plurality of poses as training data, to produce a trained neural network.

### Claim 2 (depends on 1)

The method of claim 1, wherein (A)(1) further comprises receiving, as training data:
depth data representing, for each of the plurality of training images I, depths in the three-dimensional scene represented by the training image; and
wherein (A)(2) comprises training the neural network using the plurality of training images I, the depth data, and the plurality of poses as training data, to produce the trained neural network.

### Claim 3 (depends on 2)

The method of claim 2:
wherein the depth data comprises a plurality of aligned depth images D comprising, for each of the plurality of training images I, a corresponding aligned depth image representing a depth map of the three-dimensional scene represented by the training image; and
wherein (A)(2) comprises training the neural network using the plurality of training images I, the plurality of aligned depth images D, and the plurality of poses as training data, to produce the trained neural network.

### Claim 4 (depends on 2)

The method of claim 2:
wherein (A)(1) further comprises receiving, as training data, a plurality of grasp successes indicating whether a plurality of attempts by the robot's end effector to grasp an object in the three-dimensional environment were successful; and
wherein (A)(2) comprises training the neural network using the plurality of training images I, the depth data, the plurality of poses, and the plurality of grasp successes as training data, to produce the trained neural network.

### Claim 5 (depends on 1)

The method of claim 1:
wherein (A)(1) further comprises receiving, as training data, a plurality of grasp successes indicating whether a plurality of attempts by the robot's end effector to grasp an object in the three-dimensional environment were successful; and
wherein (A)(2) comprises training the neural network using the plurality of training images I, the plurality of poses, and the plurality of grasp successes as training data, to produce the trained neural network.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
(B) before (A), generating the training data, comprising, for each of a plurality of attempts A corresponding to a plurality of target points T in the input image:
(B)(1) controlling the robot's end effector to move into the three-dimensional environment to target a point, in the three dimensional environment, corresponding to the target point T;
(B)(2) identifying a pose of the robot's end effector at a time when the robot's end effector has stopped moving in the three-dimensional environment; and
(B)(3) generating a training image of the three-dimensional environment at the time when the robot's end effector has stopped moving in the three-dimensional environment;
thereby generating the training data, the training data comprising the plurality of training images I, and the plurality of poses of the robot's end effector.

### Claim 7 (depends on 6)

The method of claim 6, wherein (B) further comprises:
(B)(4) generating an aligned depth image of the three-dimensional environment at the time when the robot's end effector has stopped moving in the three-dimensional environment; and
thereby generating the plurality of aligned depth images D.

### Claim 8 (depends on 6)

The method of claim 6, wherein (B)(1) comprises controlling the robot's end effector to move into the three-dimensional environment to target a point, in the three dimensional environment, corresponding to the target point T, and controlling the robot's end effector to attempt to grasp an object at the point in the three-dimensional environment.

### Claim 9 (depends on 8)

The method of claim 8, wherein (B) further comprises:
(B)(5) generating a grasp success indicating whether the attempt to grasp the object at the point in the three-dimensional environment was successful.

### Claim 10 (depends on 6)

The method of claim 6, wherein (B)(1) comprises:
controlling the robot's end effector to move into the three dimensional environment to target a point, corresponding to the target point T; and
identifying a point, in the three dimensional environment, at which the robot's end effector makes contact with an object in the three dimensional environment.

### Claim 11 (depends on 10)

The method of claim 10, wherein (B)(2) comprises identifying a pose of the robot's end effector at the time when the robot's end effector makes contact with the object in the three dimensional environment.

### Claim 12 (depends on 6)

The method of claim 6, wherein (B)(1) comprises:
(B)(1)(a) controlling the robot's end effector to target the point, in the three dimensional environment, corresponding to the target point T, at a first speed;
(B)(1)(b) determining that the robot's end effector is within at most a minimum distance from an object in the three dimensional environment; and
(B)(1)(c) in response to the determination of (B)(1)(b), controlling the robot's end effector to target the point, in the three dimensional environment, corresponding to the target point T, at a second speed, wherein the first speed is greater than the second speed.

### Claim 13

A system comprising at least one non-transitory computer-readable medium having computer program instructions stored thereon, the computer program instructions being executable by at least one computer processor to perform a method, the method comprising:
(A) training a neural network to predict, for each of a plurality of pixels P in an input image, a depth that a robot's end effector would reach in a three-dimensional environment if the robot's end effector moved into the three-dimensional environment to target a projection of the pixel P onto the three-dimensional environment, the method comprising:
(A)(1) receiving, as training data:
a plurality of training images I, each representing the three-dimensional scene at a corresponding time when the robot's end effector is at a corresponding stopping point in the three-dimensional scene; and
a plurality of poses of the robot's end effector, each representing a corresponding pose of the robot's end effector at a corresponding one of the stopping points; and
(A)(2) training the neural network using the plurality of training images I, and the plurality of poses as training data, to produce a trained neural network.

### Claim 14 (depends on 13)

The system of claim 13, wherein (A)(1) further comprises receiving, as training data:
depth data representing, for each of the plurality of training images I, depths in the three-dimensional scene represented by the training image; and
wherein (A)(2) comprises training the neural network using the plurality of training images I, the depth data, and the plurality of poses as training data, to produce the trained neural network.

### Claim 15 (depends on 14)

The system of claim 14:
wherein the depth data comprises a plurality of aligned depth images D comprising, for each of the plurality of training images I, a corresponding aligned depth image representing a depth map of the three-dimensional scene represented by the training image; and
wherein (A)(2) comprises training the neural network using the plurality of training images I, the plurality of aligned depth images D, and the plurality of poses as training data, to produce the trained neural network.

### Claim 16 (depends on 14)

The system of claim 14:
wherein (A)(1) further comprises receiving, as training data, a plurality of grasp successes indicating whether a plurality of attempts by the robot's end effector to grasp an object in the three-dimensional environment were successful; and
wherein (A)(2) comprises training the neural network using the plurality of training images I, the depth data, the plurality of poses, and the plurality of grasp successes as training data, to produce the trained neural network.

### Claim 17 (depends on 13)

The system of claim 13:
wherein (A)(1) further comprises receiving, as training data, a plurality of grasp successes indicating whether a plurality of attempts by the robot's end effector to grasp an object in the three-dimensional environment were successful; and
wherein (A)(2) comprises training the neural network using the plurality of training images I, the plurality of poses, and the plurality of grasp successes as training data, to produce the trained neural network.

### Claim 18 (depends on 13)

The system of claim 13, wherein the method further comprises:
(B) before (A), generating the training data, comprising, for each of a plurality of attempts A corresponding to a plurality of target points T in the input image:
(B)(1) controlling the robot's end effector to move into the three-dimensional environment to target a point, in the three dimensional environment, corresponding to the target point T;
(B)(2) identifying a pose of the robot's end effector at a time when the robot's end effector has stopped moving in the three-dimensional environment; and
(B)(3) generating a training image of the three-dimensional environment at the time when the robot's end effector has stopped moving in the three-dimensional environment;
thereby generating the training data, the training data comprising the plurality of training images I, and the plurality of poses of the robot's end effector.

### Claim 19 (depends on 18)

The system of claim 18, wherein (B) further comprises:
(B)(4) generating an aligned depth image of the three-dimensional environment at the time when the robot's end effector has stopped moving in the three-dimensional environment; and
thereby generating the plurality of aligned depth images D.

### Claim 20 (depends on 18)

The system of claim 18, wherein (B)(1) comprises controlling the robot's end effector to move into the three-dimensional environment to target a point, in the three dimensional environment, corresponding to the target point T, and controlling the robot's end effector to attempt to grasp an object at the point in the three-dimensional environment.

### Claim 21 (depends on 20)

The system of claim 20, wherein (B) further comprises:
(B)(5) generating a grasp success indicating whether the attempt to grasp the object at the point in the three-dimensional environment was successful.

### Claim 22 (depends on 6)

The system of claim 6, wherein (B)(1) comprises:
controlling the robot's end effector to move into the three dimensional environment to target a point, corresponding to the target point T; and
identifying a point, in the three dimensional environment, at which the robot's end effector makes contact with an object in the three dimensional environment.

### Claim 23 (depends on 10)

The system of claim 10, wherein (B)(2) comprises identifying a pose of the robot's end effector at the time when the robot's end effector makes contact with the object in the three dimensional environment.

### Claim 24 (depends on 6)

The system of claim 6, wherein (B)(1) comprises:
(B)(1)(a) controlling the robot's end effector to target the point, in the three dimensional environment, corresponding to the target point T, at a first speed;
(B)(1)(b) determining that the robot's end effector is within at most a minimum distance from an object in the three dimensional environment; and
(B)(1)(c) in response to the determination of (B)(1)(b), controlling the robot's end effector to target the point, in the three dimensional environment, corresponding to the target point T, at a second speed, wherein the first speed is greater than the second speed.
