# US 12,350,840 B2 · Region-based grasp generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,350,840 B2
- Document ID: US12350840B2
- Status: Granted
- Kind: B2
- Application number: 17/651,485
- Filed: 2022-02-17
- Published: 2023-08-17
- Granted: 2025-07-08
- Earliest claimed: 2022-02-17
- Assignees: Fanuc Corporation
- Applicants: Fanuc Corporation
- Inventors: Yongxiang Fan; Chi-Keng Tsai; Tetsuaki Kato
- Web page: https://patentgrep.com/patent/US12350840B2

## CPC

- B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems}
- B25J 9/1605: {Simulation of manipulator lay-out, design, modelling of manipulator}
- B25J 9/1612: {characterised by the hand, wrist, grip control}
- B25J 9/1697: {Vision controlled systems}
- B25J 13/08: by means of sensing devices, e.g. viewing or touching devices
- B25J 15/08: having finger members (B25J15/02, B25J15/04 take precedence)
- G05B 2219/35134: 3-D cad-cam
- G05B 2219/39001: Robot, manipulator control
- G05B 2219/39466: Hand, gripper, end effector of manipulator
- G05B 2219/39514: Stability of grasped objects
- G05B 2219/39536: Planning of hand motion, grasping
- G05B 2219/39539: Plan hand shape
- G05B 2219/39542: Plan grasp points, grip matrix and initial grasp force
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A region-based robotic grasp generation technique for machine tending or bin picking applications. Part and gripper geometry are provided as inputs, typically from CAD files, along with gripper kinematics. A human user defines one or more target grasp regions on the part, using a graphical user interface displaying the part geometry. The target grasp regions are identified by the user based on the user's knowledge of how the part may be grasped to ensure that the part can be subsequently placed in a proper destination pose. For each of the target grasp regions, an optimization solver is used to compute a plurality of quality grasps with stable surface contact between the part and the gripper, and no part-gripper interference. The computed grasps for each target grasp region are placed in a grasp database which is used by a robot in actual bin picking operations.

## Claims

### Claim 1

A method for generating a grasp database for use by a robotic grasping system, said method comprising:
providing an object model including three-dimensional (3D) shape data for an object to be grasped, and a gripper model including 3D shape data and actuation parameters for a gripper;
receiving input specifications from a human user, including definition of one or more target grasp regions on the object, and a required number of grasps to compute for each of the target grasp regions, where each target grasp region is a three-dimensional (3D) subpart of the object defined by techniques including selecting a component of the object model using a graphical user interface (GUI) displaying the object model and selecting a 3D volumetric sub-region of one or more components of the object model using the GUI displaying the object model;
selecting one of the target grasp regions on the object as defined by the user;
performing an iterative optimization for the selected target grasp region, using a computer having a processor and memory, where the iterative optimization converges to a grasp having a locally-maximum quality while satisfying constraint equations which define surface contact between the gripper and the selected target grasp region of the object without gripper-object interference;
outputting data defining the grasp to the grasp database;
repeatedly performing the iterative optimization and outputting the data defining the grasp until the required number of grasps is met; and
selecting another one of the target grasp regions and repeatedly performing the iterative optimization to compute the required number of grasps until all of the target grasp regions are used.

### Claim 2 (depends on 1)

The method according to claim 1 wherein performing an iterative optimization includes randomly initializing a gripper position relative to the selected target grasp region on the object for a first iteration of the optimization for each of the required number of grasps.

### Claim 3 (depends on 1)

The method according to claim 1 further comprising using the grasp database during live robotic operations to identify a target object to grasp from a container of objects, including estimating object poses by comparing depth images from a 3D camera to the object model, selecting the target object by mapping grasps from the grasp database onto the estimated object poses, and providing target object grasp data to a robot controller which instructs a robot fitted with the gripper to grasp and move the target object.

### Claim 4 (depends on 3)

The method according to claim 3 wherein the target object is placed by the robot in a destination pose, and the one or more target grasp regions are chosen by the human user to enable placement of the target object in the destination pose without interference by the gripper.

### Claim 5 (depends on 1)

The method according to claim 1 wherein the data defining the grasp includes 3D coordinates of a grasp target point, an angle of approach to be followed by the gripper, a gripper angle of rotation, a gripper width and a grasp quality value.

### Claim 6 (depends on 1)

The method according to claim 1 wherein the iterative optimization is formulated including a gripper surface defined in terms of sampling points on surfaces of the gripper model, and an object surface defined in terms of sampling points on surfaces of the object model in the target grasp region, including a nearest neighbor point on the object surface corresponding with each of the sampling points on the gripper surface.

### Claim 7 (depends on 6)

The method according to claim 6 wherein the gripper surface is transformed at each iteration by a transformation including a rotation, a translation and a change in gripper width.

### Claim 8 (depends on 7)

The method according to claim 7 wherein the iterative optimization is formulated including an objective function which maximizes a grasp quality defined based on a total distance between each sampling point on the gripper surface and its corresponding nearest neighbor point on the object surface in the target grasp region, where the constraint equations are treated as penalty functions whereby constraint violations reduce the grasp quality, and each iteration computes the transformation in one step using a least square linear algebra computation.

### Claim 9 (depends on 6)

The method according to claim 6 wherein the iterative optimization is formulated including a constraint equation which specifies that the gripper surface does not penetrate the object surface.

### Claim 10 (depends on 1)

The method according to claim 1 wherein definition of the one or more target grasp regions selected by the user includes using the GUI to select geometric elements of the object model, including one or more surface elements or solid elements.

### Claim 11

A method for generating a grasp database for use by a robotic grasping system, said method comprising:
defining one or more target grasp regions on a three-dimensional (3D) object model of an object to be grasped, where each target grasp region is a three-dimensional (3D) subpart of the object defined by techniques including selecting a component of the object model using a graphical user interface (GUI) displaying the object model and selecting a 3D volumetric sub-region of one or more components of the object model using the GUI displaying the object model; and
for each of the target grasp regions, computing a predefined required number of grasps, using a computer having a processor and memory, using an iterative optimization which converges to a grasp meeting a quality threshold while satisfying constraint equations which define surface contact between the gripper and the target grasp region of the object without gripper-object interference, and outputting each grasp to the grasp database.

### Claim 12

A robotic grasp generation system comprising:
an interface device for receiving input from a human user; and
a computer having a processor and memory, said computer communicating with the interface device and configured to generate a grasp database,
where the human user provides, using the interface device, an object model including three-dimensional (3D) shape data for an object to be grasped, and a gripper model including 3D shape data and actuation parameters for a gripper,
and the human user provides input specifications including definition of one or more target grasp regions on the object, and a required number of grasps to compute for each of the target grasp regions, where each target grasp region is a three-dimensional (3D) subpart of the object defined by techniques including selecting a component of the object model using graphical user interface (GUI) software displaying the object model on the interface device and selecting a 3D volumetric sub-region of one or more components of the object model using the GUI software displaying the object model,
and where the computer is configured to generate a grasp database, including;
selecting one of the target grasp regions on the object as defined by the user;
performing an iterative optimization for the selected target grasp region, where the iterative optimization converges to a grasp having a locally-maximum quality while satisfying constraint equations which define surface contact between the gripper and the selected target grasp region of the object without gripper-object interference;
outputting data defining the grasp to the grasp database;
repeatedly performing the iterative optimization and outputting the data defining the grasp until the required number of grasps is met; and
selecting another one of the target grasp regions and repeatedly performing the iterative optimization to compute the required number of grasps until all of the target grasp regions are used.

### Claim 13 (depends on 12)

The system according to claim 12 further comprising:
a robot controller;
a 3D camera providing depth images of a container of objects during live robotic operations, where the computer or the robot controller identifies a target object to grasp from the container of objects, including estimating object poses by comparing the depth images to the object model, and selecting the target object by mapping grasps from the grasp database onto the object poses; and
a robot fitted with the gripper to grasp and move the target object based on commands from the controller.

### Claim 14 (depends on 13)

The system according to claim 13 wherein the target object is placed by the robot in a destination pose, and the one or more target grasp regions are chosen by the human user to enable placement of the target object in the destination pose without interference by the gripper.

### Claim 15 (depends on 12)

The system according to claim 12 wherein the data defining the grasp includes 3D coordinates of a grasp target point, an angle of approach to be followed by the gripper, a gripper angle of rotation, a gripper width and a grasp quality value.

### Claim 16 (depends on 12)

The system according to claim 12 wherein performing an iterative optimization includes randomly initializing a gripper position relative to the selected target grasp region on the object for a first iteration of the optimization for each of the required number of grasps.

### Claim 17 (depends on 12)

The system according to claim 12 wherein the iterative optimization is formulated including a gripper surface defined in terms of sampling points on surfaces of the gripper model, and an object surface defined in terms of sampling points on surfaces of the object model in the target grasp region, including a nearest neighbor point on the object surface corresponding with each of the sampling points on the gripper surface, and where the gripper surface is transformed at each iteration by a transformation including a rotation, a translation and a change in gripper width.

### Claim 18 (depends on 17)

The system according to claim 17 wherein the iterative optimization is formulated including an objective function which maximizes a grasp quality defined based on a total distance between each sampling point on the gripper surface and its corresponding nearest neighbor point on the object surface in the target grasp region, where the constraint equations are treated as penalty functions whereby constraint violations reduce the grasp quality, and each iteration computes the transformation in one step using a least square linear algebra computation.

### Claim 19 (depends on 17)

The system according to claim 17 wherein the iterative optimization is formulated including a constraint equation which specifies that the gripper surface does not penetrate the object surface.

### Claim 20 (depends on 12)

The system according to claim 12 wherein definition of the one or more target grasp regions selected by the user includes using the GUI software on the interface device to select geometric elements of the object model, including one or more surface elements or solid elements.