# US 2024/0198515 A1 · Transformation for covariate shift of grasp neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0198515 A1
- Document ID: US20240198515A1
- Status: Published application
- Kind: A1
- Application number: 18/555,780
- Filed: 2021-05-25
- Published: 2024-06-20
- Earliest claimed: 2021-05-25
- Assignees: Siemens Aktiengesellschaft
- Applicants: Siemens Aktiengesellschaft
- Inventors: Juan L. Aparicio Ojea; Heiko Claussen; Ines Ugalde Diaz; Gokul Narayanan Sathya Narayanan; Eugen Solowjow; Chengtao Wen; Wei Xi Xia; Yash Shahapurkar; Shashank Tamaskar
- Web page: https://patentgrep.com/patent/US20240198515A1

## CPC

- B25J 9/1612: {characterised by the hand, wrist, grip control}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G05B 19/4155: characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program
- G06T 7/0004: {Industrial image inspection}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/82: using neural networks
- G05B 2219/39543: Recognize object and plan hand shapes in grasping movements
- G05B 2219/40269: Naturally compliant robot arm
- G05B 2219/40584: Camera, non-contact sensor mounted on wrist, indep from gripper
- G05B 2219/40613: Camera, laser scanner on end effector, hand eye manipulator, local
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A covariate shift generally refers to the change of the distribution of the input data (e.g., noise distribution) between the training and inference regimes. Such covariate shifts can degrade the performance grasping neural networks, and thus robotic grasping operations. As described herein, an output of a grasp neural network can be transformed, so as to determine appropriate locations on a given object for a robot or autonomous machine to grasp.

## Claims

### Claim 1

A method of grasping an object, the method comprising:
capturing an image of the object within a physical environment, each of the object and the physical environment defining a plurality of locations, the image defining a plurality of pixels representative of the plurality of locations of the object and the physical environment;
sending the image of the object within the physical environment to a neural network;
based on the image, the neural network generating an output that defines respective grasp scores associated with the plurality of pixels, the grasp scores indicating a respective grasp quality at the locations of the object represented by the respective pixel;
transforming the output of the neural network;
based on transforming the output of the neural network, selecting a grasp location of the plurality of locations of the object;
grasping the object, by an autonomous machine, at the grasp location.

### Claim 2 (depends on 1)

The method as recited in claim 1, wherein transforming the output of the neural network further comprises:
identifying a first grasp score of the grasp scores that indicates a high grasp quality as compared to the other grasp scores, the first grasp score associated with a first location of the plurality of locations; and
rejecting the first grasp score that indicates the high grasp quality, such that the grasp location that is selected is associated with one of the grasp scores other than the first grasp score.

### Claim 3 (depends on 2)

The method as recited in claim 2, wherein rejecting the first grasp score comprises:
determining that the first location defines a point on the physical environment outside of the object.

### Claim 4 (depends on 2)

The method as recited in claim 2, wherein rejecting the first grasp score further comprises:
sorting the grasp scores in an order from high grasp quality to low grasp quality, so as to define a sorted list of grasp scores that defines at least one grasp score proximate to the first grasp score; and
determining that the respective locations associated with the at least one grasp score proximate to the first grasp score are greater than a predetermined distance away from the first location associated with the first grasp score.

### Claim 5 (depends on 2)

The method as recited in claim 2, wherein the object defines at least one edge, the first grasp score is associated with a first location, and rejecting the first grasp score further comprises:
determining a first distance from the first location to the at least one edge;
comparing the first distance to a predetermined threshold; and
determining that the first distance is less than the predetermined threshold.

### Claim 6 (depends on 5)

The method as recited in claim 5, wherein the autonomous machine comprises an end effector configured to grasp the object, and the predetermined threshold is based on the end effector.

### Claim 7 (depends on 6)

The method as recited in claim 6, wherein the predetermined threshold is based on a size of the end effector.

### Claim 8 (depends on 7)

The method as recited in claim 7, wherein the end effector includes a suction cup having a diameter, and the predetermined threshold is based on the diameter.

### Claim 9 (depends on 2)

The method as recited in claim 2, wherein transforming the output of the neural network further comprises:
identifying a first grasp score that indicates a high grasp quality as compared to the other grasp scores, the first grasp score associated with a first location of the plurality of locations;
determining a first distance between an edge of the object and the first location;
based on the first distance, determining a second distance; and
selecting the grasp location such that the grasp location is the second distance away from the first location.

### Claim 10

A method for determining grasp points associated with an autonomous machine that defines an end effector configured to grasp objects, the method comprising:
training a neural network on images of a plurality of objects;
sending an image of a new object to the neural network, the new object not one of the plurality of objects on which the neural network is trained;
based on the image and the end effector, the neural network generating an output that identifies grasp locations on the object, the output defining a covariate shift; and
transforming the output so as to eliminate the covariate shift.

### Claim 11 (depends on 10)

The method of claim 10, wherein transforming the output further comprising:
changing at least one of the grasp locations identified by the output of the neural network.

### Claim 12

An autonomous system comprising:
an autonomous machine defining an end effector configured to grasp an object within a physical environment, each of the object and the physical environment defining a plurality of locations;
a sensor configured to capture an image of an object within the physical environment, the image defining a plurality of pixels representative of the plurality of locations of the object and the physical environment;
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the autonomous system to:
based on the image, generate an output that defines respective grasp scores associated with the plurality of pixels, the grasp scores indicating a respective grasp quality at the locations of the object represented by the respective pixel;
transform the output;
based on transforming the output, select a grasp location of the plurality of locations of the object; and
send an instruction to the autonomous machine to grasp the object at the grasp location,
wherein the end effector is further configured to, based on the instruction, grasp the object at the grasp location.

### Claim 13 (depends on 12)

The autonomous system of claim 12, the memory further storing instructions that, when executed by the one or more processors, further cause the autonomous system to:
identify a first grasp score of the grasp scores that indicates a high grasp quality as compared to the other grasp scores, the first grasp score associated with a first location of the plurality of locations; and
reject the first grasp score that indicates the high grasp quality, such that the grasp location that is selected is associated with one of the grasp scores other than the first grasp score.

### Claim 14 (depends on 13)

The autonomous system of claim 13, the memory further storing instructions that, when executed by the one or more processors, further cause the autonomous system to:
determining that the first location defines a point on the physical environment outside of the object.

### Claim 15 (depends on 13)

The autonomous system of claim 13, the memory further storing instructions that, when executed by the one or more processors, further cause the autonomous system to:
sort the grasp scores in an order from high grasp quality to low grasp quality, so as to define a sorted list of grasp scores that defines at least one grasp score proximate to the first grasp score; and
determining that the respective locations associated with the at least one grasp score proximate to the first grasp score are greater than a predetermined distance away from the first location associated with the first grasp score.

### Claim 16 (depends on 1)

A non-transitory computer-readable storage medium including instructions that, when processed by a computing system, configure the computing system to perform the method according to claim 1.
