# US 12,377,549 B2 · System and method for determining a grasping hand model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,377,549 B2
- Document ID: US12377549B2
- Status: Granted
- Kind: B2
- Application number: 17/833,460
- Filed: 2022-06-06
- Published: 2022-12-22
- Granted: 2025-08-05
- Earliest claimed: 2020-06-09
- Assignees: Naver Corporation
- Applicants: Naver Corporation; Naver Labs Corporation
- Inventors: Francesc Moreno Noguer; Guillem Alenyà Ribas; Enric Corona Puyane; Albert Pumarola Peris; Grégory Rogez
- Web page: https://patentgrep.com/patent/US12377549B2

## CPC

- B25J 9/1612: {characterised by the hand, wrist, grip control}
- B25J 9/1697: {Vision controlled systems}
- B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor}
- G05B 2219/39536: Planning of hand motion, grasping
- G05B 2219/39546: Map human grasps to manipulator grasps
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Method for determining a grasping hand model suitable for grasping an object by receiving an image including at least one object; obtaining an object model estimating a pose and shape of the object from the image of the object; selecting a grasp class from a set of grasp classes by means of a neural network, with a cross entropy loss, thus, obtaining a set of parameters defining a coarse grasping hand model; refining the coarse grasping hand model, by minimizing loss functions referring to the parameters of the hand model for obtaining an operable grasping hand model while minimizing the distance between the finger of the hand model and the surface of the object and preventing interpenetration; and obtaining a mesh of the hand represented by the enhanced set of parameters.

## Claims

### Claim 1

A system for determining parameters of a hand model suitable for grasping an object, comprising:
a first neural network for segmenting the object in an image;
a second neural network for predicting parameters of the hand model that define a pose for grasping the segmented object; and
a third neural network for refining the predicted parameters of the hand model to fit the segmented object by maximizing a number of contact points between the object in the image and the hand model while minimizing interpenetration;
wherein the refined predicted parameters of the hand model represent how a hand may grasp the object.

### Claim 2 (depends on 1)

The system according to claim 1, wherein said first neural network segments the object in an image by estimating a pose and a 3D shape of the object from the image of the object.

### Claim 3 (depends on 1)

The system according to claim 1, wherein said second neural network predicts parameters of the hand model by predicting a grasp class from a set of grasp classes to obtain a set of parameters defining a coarse grasping hand model.

### Claim 4 (depends on 1)

The system according to claim 1, wherein said third neural network refines the predicted parameters of the hand model by minimizing loss functions referring to the parameters of the hand model for obtaining an operable grasping hand model while minimizing a distance between the fingers of the hand model and a surface of the object and preventing interpenetration.

### Claim 5 (depends on 1)

The system according to claim 1, wherein said second neural network is a Convolutional Neural Network, with a cross entropy loss Lclass defined as:
Lclass=Σc∈KCo,c log(1−Po,c)
wherein C represents a grasp type for the particular object (o), c represents the grasp classes among the K possible grasps classes, and P represents pose predictions for the particular object (o).

### Claim 6 (depends on 1)

The system according to claim 1, wherein said third neural network obtains a representation of a hand grasping the object by using the refined predicted parameters of the hand model, the representation being a mesh of the refined hand model.

### Claim 7 (depends on 1)

The system according to claim 1, wherein said third neural network uses a MANO model, being a 51 degrees of freedom model of a possible human hand, for refining the predicted parameters of the hand model.

### Claim 8 (depends on 1)

The system according to claim 1, wherein said third neural network evaluates the grasping hand model by calculating at least one evaluating metric of an analytical grasp metric, which computes an approximation of the minimum force to be applied to break the grasp stability; an average number of contact fingers, wherein numerous contact points between hand and object favor a strong grasp; a hand-object interpenetration volume, wherein object and hand are voxelized, and the volume shared by both 3D models is computed; a simulation displacement of the object mesh subjected to gravity; and a percentage of graspable objects for which an operable grasp could be predicted, being an operable grasp the one with at least two contact points and no interpenetration.

### Claim 9 (depends on 8)

The system according to claim 8, wherein said third neural network (a) randomly rotates an object model; (b) obtains a grasping hand model for each rotated object model; (c) evaluates each rotated grasping hand model using evaluating metrics; and (d) selects the rotated grasping hand models having the highest score.

### Claim 10 (depends on 1)

The system according to claim 1, wherein said second neural network estimates a pose and shape of the object by using an object reconstruction phase for obtaining a cloud of points representing the object form the obtained image.

### Claim 11 (depends on 1)

The system according to claim 1, wherein said image includes more than one object;
said first neural network segmenting each object in said image and estimating a 3D shape of each segmented object;
said second neural network predicting parameters of the hand model that define a pose for grasping each segmented object;
said third neural network refining the predicted parameters of the hand model to fit each segmented object.

### Claim 12 (depends on 1)

The system according to claim 1, wherein said second neural network selects a grasp class by utilizing a phase of predicting an increment of translation and rotation of the hand model and a modified coarse configuration of the hand model.

### Claim 13 (depends on 1)

The system according to claim 1, wherein said third neural network refines the coarse grasping model by:
(a) selecting at least one articulation (i) of the hand model;
(b) calculating an arc (Ai) between a finger (j) of the hand model and close object vertices (O),
Dθ←mini(mink(∥Aiθ,Ok∥2))
(c) estimating the angle the finger needs to be rotated to collide with the object, rotating the articulation for minimizing the arc, thus, reducing the distance between the hand model and the object vertices, including a hyperparameter for controlling the interpenetration of the hand model into the object,
γj′←arg minθDθ+δ, ∀θs.t. Dθ<td
(d) defining the following loss functions:
L
a
⁢
r
⁢
c
=
1
❘
"\[LeftBracketingBar]"
J
❘
"\[RightBracketingBar]"
⁢
∑
j
∈
J
D
θ
j
L
γ
←
∑
j
J
❘
"\[LeftBracketingBar]"
❘
"\[LeftBracketingBar]"
γ
j
′
-
γ
j
❘
"\[RightBracketingBar]"
❘
"\[RightBracketingBar]"
2
;
and
(e) minimizing the loss functions defined.

### Claim 14 (depends on 13)

The system according to claim 13, wherein said third neural network refines the coarse grasping model for each articulation sequentially from the knuckle to the tip for each finger.

### Claim 15 (depends on 1)

The system according to claim 1, wherein said third neural network refines the coarse grasping model by minimizing a loss function from a distance between the hand vertices and the target object, wherein is considered that there is a contact when the distance is below to 2 mm, defined by
L
cont
=
1
❘
"\[LeftBracketingBar]"
V
cont
❘
"\[RightBracketingBar]"
⁢
∑
v
∈
V
cont
min
k
❘
"\[LeftBracketingBar]"
❘
"\[LeftBracketingBar]"
v
,
O
k
t
❘
"\[RightBracketingBar]"
❘
"\[RightBracketingBar]"
2
.

### Claim 16 (depends on 1)

The system according to claim 1, wherein said third neural network refines the coarse grasping model by minimizing a loss function from a distance of interpenetration between a vertex of the hand model and the object, defined by
L
int
=
1
❘
"\[LeftBracketingBar]"
V
i
❘
"\[RightBracketingBar]"
⁢
∑
j
❘
"\[LeftBracketingBar]"
O
❘
"\[RightBracketingBar]"
∑
v
∈
V
i
min
k

v
,
O
k
j

2
.

### Claim 17 (depends on 1)

The system according to claim 1, wherein said third neural network refines the coarse grasping model by minimizing a loss function from a distance below a table plane, between a vertex of the hand model and the table plane, wherein the distance is favored to be positive, defined by Lp=Σvvmin (0, |(v−pp)·vp|).

### Claim 18 (depends on 1)

The system according to claim 1, wherein said third neural network refines the coarse grasping model by minimizing a loss function from an adversarial loss function, using a Wasserstein loss including a gradient penalty loss, defined by Ladv=−EH,R,T˜p(H,R,T)[D(G(I))]+EH,R,T˜p(H,R,T)[D(H*, R*, T*)].

### Claim 19 (depends on 1)

The system according to claim 1, wherein the hand is a human hand.

### Claim 20

A method for determining parameters of a hand model suitable for grasping an object, comprising:
(a) receiving an object segmented in an image;
(b) predicting, with a first neural network, parameters of the hand model that define a pose for grasping the segmented object; and
(c) refining, with a second neural network, the predicted parameters of the hand model to fit the segmented object by maximizing a number of contact points between the object in the image and the hand model while minimizing interpenetration;
wherein the refined predicted parameters of the hand model represent how a hand may grasp the object.

### Claim 21 (depends on 20)

The method according to claim 20, wherein said receiving is performed using a third neural network.

### Claim 22 (depends on 20)

The method according to claim 20, wherein (b) predicts parameters of the hand model by predicting a grasp class from a set of grasp classes to obtain a set of parameters defining a coarse grasping hand model.

### Claim 23 (depends on 20)

The method according to claim 20, wherein (c) refines the predicted parameters of the hand model by minimizing loss functions referring to the parameters of the hand model for obtaining an operable grasping hand model while minimizing the distance between the fingers of the hand model and the surface of the object and preventing interpenetration.

### Claim 24 (depends on 20)

The method according to claim 20, further comprising:
(d) obtaining a representation of a hand grasping the object by using the refined hand model.

### Claim 25 (depends on 20)

The method according to claim 20, wherein the first neural network is a Convolutional Neural Network, with a cross entropy loss Lclass defined as:
Lclass=Σc∈KCo,clog (1−Po,c);
wherein C represents a grasp type for the particular object (o), c represents the grasp classes among the K possible grasps classes, and P represents pose predictions for the particular object (o).

### Claim 26 (depends on 24)

The method according to claim 24, wherein the representation obtained in (d) is a mesh of the refined hand model.

### Claim 27 (depends on 20)

The method according to claim 20, wherein the hand model is represented by using a MANO model, being a 51 degrees of freedom model of a possible human hand.

### Claim 28 (depends on 20)

The method according to claim 20, further comprising:
(d) evaluating the grasping hand model obtained by calculating at least one evaluating metric of an analytical grasp metric, which computes an approximation of the minimum force to be applied to break the grasp stability; an average number of contact fingers, wherein numerous contact points between hand and object favor a strong grasp; a hand-object interpenetration volume, wherein object and hand are voxelized, and the volume shared by both 3D models is computed; a simulation displacement of the object mesh subjected to gravity; and a percentage of graspable objects for which an operable grasp could be predicted, being an operable grasp the one with at least two contact points and no interpenetration.

### Claim 29 (depends on 28)

The method according to claim 28, further comprising:
(e) randomly rotating an object model;
(f) obtaining a grasping hand model for each rotated object model;
(g) evaluating each rotated grasping hand model using evaluating metrics; and
(h) selecting the rotated grasping hand models having the highest score.

### Claim 30 (depends on 21)

The method according to claim 21, wherein estimating a pose and shape of the object comprises an object reconstruction phase for obtaining a cloud of points representing the object form the obtained image.

### Claim 31 (depends on 20)

The method according to claim 20, wherein the image comprises more than one object and the method further comprises the step of repeating (a) to (c) for each object in the image, wherein the objects are known.

### Claim 32 (depends on 20)

The method according to claim 20, wherein (b) selects a grasp class by utilizing a phase of predicting an increment of translation and rotation of the hand model and a modified coarse configuration of the hand model.

### Claim 33 (depends on 20)

The method according to claim 20, wherein (c) includes:
(c1) selecting at least one articulation (i) of the hand model;
(c2) calculating an arc (Ai) between a finger (j) of the hand model and close object vertices (O),
Dθ←mini(mink(∥i Aiθ, Ok∥2));
(c3) estimating the angle the finger needs to be rotated to collide with the object, rotating the articulation for minimizing the arc, thus, reducing the distance between the hand model and the object vertices, including a hyperparameter for controlling the interpenetration of the hand model into the object,
γj′←arg minθDθ+δ,∀θs.t. Dθ<td;
(c4) defining the following loss functions:
L
arc
=
1
❘
"\[LeftBracketingBar]"
J
❘
"\[RightBracketingBar]"
⁢
∑
j
∈
J
D
θ
j
L
γ
←
∑
j
J
⁢

γ
j
′
-
γ
j

2
;
and
(c5) minimizing the loss functions defined.

### Claim 34 (depends on 33)

The method according to claim 33, wherein (c) further includes (c6) repeating phases (c2) and (c3) for each articulation sequentially from the knuckle to the tip for each finger.

### Claim 35 (depends on 20)

The method according to claim 20, wherein said (c) further comprises minimizing a loss function from a distance between the hand vertices and the target object, wherein is considered that there is a contact when the distance is below to 2 mm, defined by
L
cont
=
1
❘
"\[LeftBracketingBar]"
V
cont
❘
"\[RightBracketingBar]"
⁢
∑
v
∈
V
cont
min
k

v
,
O
k
t

2
.

### Claim 36 (depends on 20)

The method according to claim 20, wherein said (c) further comprises minimizing a loss function from a distance of interpenetration between a vertex of the hand model and the object, defined by
L
int
=
1
❘
"\[LeftBracketingBar]"
V
i
❘
"\[RightBracketingBar]"
⁢
∑
j
❘
"\[LeftBracketingBar]"
O
❘
"\[RightBracketingBar]"
∑
v
∈
V
i
min
k

v
,
O
k
j

2
.

### Claim 37 (depends on 20)

The method according to claim 20, wherein said (c) further comprises minimizing a loss function from a distance below a table plane, between a vertex of the hand model and the table plane, wherein the distance is favored to be positive, defined by Lp=Σvvmin (0, |(v−pp)·vp|).

### Claim 38 (depends on 20)

The method according to claim 20, wherein said (c) further comprises minimizing a loss function from an adversarial loss function, using a Wasserstein loss including a gradient penalty loss, defined by Ladv=−EH,R,T˜p(H,R,T)[D(G(I))]+EH,R,T˜p(H,R,T)[D(H*, R*, T*)].

### Claim 39 (depends on 20)

The method according to claim 20, wherein the hand is a human hand.

### Claim 40 (depends on 20)

The method according to claim 20, wherein (a) segments an object in an image by estimating a pose and shape of the object from the image of the object.