US 12,293,540 B2 · Apparatus for constructing kinematic information of robot manipulator and method therefor

Abstract

An apparatus for constructing kinematic information of a robot manipulator is provided. The apparatus includes: a robot image acquisition part for acquiring a robot image containing shape information and coordinate information of the robot manipulator; a feature detection part for detecting the type of each of a plurality of joints of the robot manipulator and the three-dimensional coordinates of the joint using a feature detection model generated through deep learning based on the robot image containing shape information and coordinate information; and a variable derivation part for deriving Denavit-Hartenberg (DH) parameters based on the type of each of the plurality of joints of the robot manipulator and the three-dimensional coordinates of the joint.

An apparatus for constructing kinematic information of a robot manipulator, the apparatus comprising:

a robot image acquisition part for acquiring a composite robot image containing shape information and three-dimensional coordinate information of the robot manipulator;
a processor comprising a feature detection part configured for detecting the type of each of a plurality of joints of the robot manipulator and the three-dimensional coordinates of the joint using a feature detection model generated through deep learning based on the composite robot image containing shape information and three-dimensional coordinate information;
the processor comprising a variable derivation part configured for deriving Denavit-Hartenberg (DH) parameters based on the type of each of the plurality of joints of the robot manipulator and the three-dimensional coordinates of the joint;
and
wherein the processor feeds the composite robot image containing shape information and three-dimensional coordinate information as input into the feature detection model generated through deep learning, and the feature detection model produces a computed value for each of a plurality of joints of the robot manipulator included in the composite robot image, including the type of the joint and the three-dimensional coordinates of the joint, by performing operations using learned weights on the shape information and three-dimensional coordinate information of the composite robot image,
wherein the feature detection part is configured to perform Denavit-Hartenberg (DH) parameter modification and robot kinematic calibration in response to a user input, by using a detailed specification of the robot manipulator;
and
wherein the processor further comprises a model generation part which provides a labeled composite robot image for training that includes the type of each of the plurality of joints of the robot manipulator and the three-dimensional coordinates of the joint, which feeds the composite robot image for training as input into a feature detection model with initialized parameters, which, once the feature detection model with initialized parameters produces a computed value for each of the plurality of joints of the robot manipulator, including the type of the joint and the three-dimensional coordinates of the joint, by performing operations using weights initialized from the composite robot image for training, produces a loss representing the difference between the computed value and a label, and which performs optimization to modify the weights of the feature detection model so as to minimize the difference between the computed value and the label, and
wherein the model generation part obtains a shape loss by using a loss function according to Equation 1:
ω
coord
⁢
∑
s
2
i
=
0
∑
B
j
=
0
1
ij
obj
[
(
dx
i
-
dx
⋀
i
)
2
+
(
dy
i
-
dy
⋀
i
)
2
]
+
ω
coord
⁢
∑
s
2
i
=
0
∑
B
j
=
0
1
ij
obj
[
(
w
i
-
w
^
i
)
2
+
(
h
i
-
h
^
i
)
2
]
+
∑
s
2
i
=
0
∑
B
j
=
0
1
ij
obj
⁢
(
C
i
-
C
^
i
)
2
+
ω
coord
⁢
∑
s
2
i
=
0
∑
B
j
=
0
1
ij
noobj
⁢
(
C
i
-
C
^
i
)
2
+
∑
s
2
i
=
0
1
i
obj
⁢
∑
c
∈
classes
(
p
i
(
c
)
-
p
^
i
(
c
)
)
2
[
Equation
⁢
1
]
wherein S represents the number of cells,
B represents the number of bounding boxes in a cell,
dx and dy represent center coordinates of a bounding box, and
w and h represent width and height of the bounding box,
C represents confidence score,
pi(c) represents the probability of an object in an ith cell belonging to a corresponding class (c), wherein i is an index representing a cell where an object is present, and j is an index representing a predicted bounding box,
ωcoord is used to reflect a higher value for a variable of the bounding box, which is a parameter for balancing loss and other losses with the coordinates (dx, dy, w, h) of the bounding box,
ωnoobj is used to reflect a higher value for a variable of the bounding box and a lower value for a variable of an area where the object is not present such that ωnoobj is a parameter for balancing between bounding boxes with or without objects,
1iobj represents that the object is in a cell and 1ijobj represents the jth bounding box is in cell i,
wherein 1ijnoobj is a complement of 1ijobj and 1ijnoobj represents the jth bounding box is in cell i, where there is no object,
wherein dxi, dvi, wi, hi, Ci, and pi(c) with a {circumflex over ( )} represent the value computed by the feature detection model:
and the model generation part obtains spatial coordinate loss by a loss function according to Equation 2;
∑
s
2
i
=
0
∑
B
j
=
0
1
ij
obj
[
(
x
i
-
x
^
i
)
2
+
(
y
i
-
y
^
i
)
2
+
(
z
i
-
z
^
i
)
2
]
[
Equation
⁢
2
]
wherein x, y, and z represent three-dimensional coordinates corresponding to the center coordinates (dx, dy) of the bounding box, wherein x, y, and z with a {circumflex over ( )} represent the value computed by the feature detection model, and the other parameters in Equation 2 are the same as in Equation 1.

234 more elements

Apparatus for constructing kinematic information of robot manipulator and method therefor

Life of this patent

Apparatus for constructing kinematic information of robot manipulator and method therefor

Life of this patent

Abstract

Claims4