# US 2021/0197389 A1 · Computer device and method for controlling robotic arm to grasp and place objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0197389 A1
- Document ID: US20210197389A1
- Status: Published application
- Kind: A1
- Application number: 17/136,934
- Filed: 2020-12-29
- Published: 2021-07-01
- Earliest claimed: 2019-12-30
- Assignees: Chiun Mai Communication Systems, Inc.
- Applicants: Chiun Mai Communication Systems, Inc.
- Inventors: TUNG-CHUN HSIEH
- Web page: https://patentgrep.com/patent/US20210197389A1

## CPC

- B25J 9/1697: {Vision controlled systems}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- G06F 18/25: Fusion techniques
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/64: Three-dimensional [3D] objects
- G05B 2219/39543: Recognize object and plan hand shapes in grasping movements
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20221: Image fusion; Image merging
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A method for controlling a robotic arm to grasp and place objects includes acquiring a plurality of sets of images each of which including an RGB image and a depth image. The RGB image and the depth image of each set are associated with each other. A plurality of fused images are obtained by fusing of depth information on each RGB image based on depth information of corresponding depth image. Once a three-dimensional map is constructed based on the plurality of fused images, a robotic arm is controlled to grasp and place objects based on the three-dimensional map.

## Claims

### Claim 1

A method of controlling a robotic arm to grasp and place objects comprising:
obtaining a plurality of sets of images, the plurality of sets of images comprising a plurality of RGB images and a plurality of depth images taken by a depth camera module of the robotic arm, and each set of the plurality of sets of images comprises one RGB image and one depth image;
associating the one RGB image with the one depth image of the each set of the plurality of sets of images;
processing the plurality of RGB images by a preset image processing algorithm;
fusing of depth information on the processed RGB images based on the depth information of the plurality of depth images to obtain a plurality of fused images;
constructing a three-dimensional map based on the plurality of fused images; and
controlling the robotic arm to grasp objects and place objects based on the three-dimensional map.

### Claim 2 (depends on 1)

The method as claimed in claim 1, wherein the processing the plurality of RGB images by a preset image processing algorithm comprising:
performing a first processing on the plurality of RGB images to obtain first processed RGB images, wherein the first processing comprises: performing feature point matching on each two adjacent RGB images of the plurality of RGB images using a SURF algorithm;
performing a second processing on the first processed RGB images to obtain second processed RGB images, wherein the second processing comprises: confirming whether the feature point matching has been correctly performed on each two adjacent RGB images of the first processed RGB images, and eliminating wrongly matched feature point; and
performing a third processing on the second processed RGB images to obtain third processed RGB images, wherein the third processing comprises: calculating a graphic angle difference of each two adjacent RGB images of the second processed RGB images using a RANSAC algorithm, and making the graphic angle of each two adjacent RGB images the same by correspondingly correcting one of the each two adjacent RGB images based on the graphic angle difference.

### Claim 3 (depends on 1)

The method as claimed in claim 1, wherein the constructing a three-dimensional map based on the plurality of fused images comprises:
calculating three-dimensional coordinates of each pixel point of each fused image of the plurality of fused images in a physical space;
associating each fused image with the three-dimensional coordinates of each pixel point of the each fused image, and stitching the plurality of fused images to obtain the three-dimensional map.

### Claim 4 (depends on 3)

The method as claimed in claim 3, wherein the three-dimensional coordinates of each pixel point p1 of the each fused image is (x1, y1, z1), z1=d/s; x1=(xx1−cx)*z1/fx; y1=(yy1−cy)*z1/fy; xx1 representing an abscissa of the pixel point p1 in the fused image, yy1 representing an ordinate of the pixel point p1 in the fused image, and d representing a vertical coordinate of the pixel point p1 in the fused image; fx representing a focal length of the RGB camera of depth camera module on an x-axis of the coordinate system, and fy representing the focal length of the RGB camera of depth camera module on a y-axis; and cx representing a distance from a center of an aperture of the RGB camera of depth camera module to the x-axis, cy representing a distance from the center of the aperture of the RGB camera of depth camera module to the y axis; and s representing a zoom value of the RGB camera of depth camera module.

### Claim 5 (depends on 4)

The method as claimed in claim 4, wherein the controlling the robotic arm to grasp objects and place objects based on the three-dimensional map comprises:
locating position coordinates of the robotic arm based on the three-dimensional map, when the three-dimensional map obtained;
obtaining first position coordinates of a target object, wherein the first position coordinates are coordinates of a current position of the target object;
controlling the robotic arm to grasp the target object based on the position coordinates of the robotic arm and the first position coordinates of the target object;
obtaining second position coordinates of the target object, wherein the second position coordinates are coordinates of a target position where the target object needs to be placed; and
controlling the robotic arm to place the target object on the target position based on the second position coordinates of the target object.

### Claim 6 (depends on 5)

The method as claimed in claim 5, wherein the controlling the robotic arm to grasp objects and place objects based on the three-dimensional map further comprises:
determining whether the robotic arm grasps the target object based on the position coordinates of the robotic arm and the first position coordinates of the target object;
recognizing the target object and measuring position coordinates of the target object when the robotic arm fails to grasp the target object; and
controlling the robotic arm to grasp the target object based on the measured position coordinates of the target object.

### Claim 7

A computer device comprising:
a storage device;
at least one processor; and
the storage device storing one or more programs, which when executed by the at least one processor, cause the at least one processor to:
obtain a plurality of sets of images, wherein the plurality of sets of images comprise a plurality of RGB images and a plurality of depth images taken by a depth camera module of a robotic arm, and each set of the plurality of sets of images comprises one RGB image and one depth image;
associate the one RGB image with the one depth image of the each set of the plurality of sets of images;
process the plurality of RGB images by a preset image processing algorithm;
fuse of depth information on the processed RGB images based on the depth information of the plurality of depth images, to obtain a plurality of fused images;
construct a three-dimensional map based on the plurality of fused images; and
control the robotic arm to grasp objects and place objects based on the three-dimensional map.

### Claim 8 (depends on 7)

The computer device as claimed in claim 7, wherein the processing the plurality of RGB images by a preset image processing algorithm comprising:
performing a first processing on the plurality of RGB images to obtain first processed RGB images, wherein the first processing comprises: performing feature point matching on each two adjacent RGB images of the plurality of RGB images using a SURF algorithm;
performing a second processing on the first processed RGB images to obtain second processed RGB images, wherein the second processing comprises: confirming whether the feature point matching has been correctly performed on each two adjacent RGB images of the first processed RGB images, and eliminating wrongly matched feature point; and
performing a third processing on the second processed RGB images to obtain third processed RGB images, wherein the third processing comprises: calculating a graphic angle difference of each two adjacent RGB images of the second processed RGB images using a RANSAC algorithm, and making the graphic angle of each two adjacent RGB images the same, by correspondingly correcting one of the each two adjacent RGB images based on the graphic angle difference.

### Claim 9 (depends on 7)

The computer device as claimed in claim 7, wherein the constructing a three-dimensional map based on the plurality of fused images comprises:
calculating three-dimensional coordinates of each pixel point of each fused image of the plurality of fused images in a physical space;
associating each fused image with the three-dimensional coordinates of each pixel point of the each fused image, and stitching the plurality of fused images to obtain the three-dimensional map.

### Claim 10 (depends on 9)

The computer device as claimed in claim 9, wherein the three-dimensional coordinates of each pixel point p1 of the each fused image is (x1, y1, z1), z1=d/s; x1=(xx1−cx)*z1/fx; y1=(yy1−cy)*z1/fy; xx1 representing an abscissa of the pixel point p1 in the fused image, yy1 representing an ordinate of the pixel point p1 in the fused image, and d representing a vertical coordinate of the pixel point p1 in the fused image; fx representing a focal length of the RGB camera of depth camera module on an x-axis of the coordinate system, and fy representing the focal length of the RGB camera of depth camera module on a y-axis; and cx representing a distance from a center of an aperture of the RGB camera of depth camera module to the x-axis, cy representing a distance from the center of the aperture of the RGB camera of depth camera module to the y axis; and s representing a zoom value of the RGB camera of depth camera module.

### Claim 11 (depends on 10)

The computer device as claimed in claim 10, wherein the controlling the robotic arm to grasp objects and place objects based on the three-dimensional map comprises:
locating position coordinates of the robotic arm based on the three-dimensional map, when the three-dimensional map obtained;
obtaining first position coordinates of a target object, wherein the first position coordinates are coordinates of a current position of the target object;
controlling the robotic arm to grasp the target object based on the position coordinates of the robotic arm and the first position coordinates of the target object;
obtaining second position coordinates of the target object, wherein the second position coordinates are coordinates of a target position where the target object needs to be placed; and
controlling the robotic arm to place the target object on the target position based on the second position coordinates of the target object.

### Claim 12 (depends on 11)

The computer device as claimed in claim 11, wherein the controlling the robotic arm to grasp objects and place objects based on the three-dimensional map further comprises:
determining whether the robotic arm grasps the target object based on the position coordinates of the robotic arm and the first position coordinates of the target object;
recognizing the target object and measuring position coordinates of the target object when the robotic arm fails to grasp the target object; and
controlling the robotic arm to grasp the target object based on the measured position coordinates of the target object.

### Claim 13

A non-transitory storage medium having instructions stored thereon, when the instructions are executed by a processor of a computer device, the processor is configured to perform a method of controlling a robotic arm to grasp and place an object, wherein the method comprises:
obtaining a plurality of sets of images, wherein the plurality of sets of images comprise a plurality of RGB images and a plurality of depth images taken by a depth camera module of the robotic arm, and each set of the plurality of sets of images comprises one RGB image and one depth image;
associating the one RGB image with the one depth image of the each set of the plurality of sets of images;
processing the plurality of RGB images by a preset image processing algorithm;
fusing of depth information on the processed RGB images based on the depth information of the plurality of depth images to obtain a plurality of fused images;
constructing a three-dimensional map based on the plurality of fused images; and
controlling the robotic arm to grasp objects and place objects based on the three-dimensional map.

### Claim 14 (depends on 13)

The non-transitory storage medium as claimed in claim 13, wherein the processing the plurality of RGB images by a preset image processing algorithm comprising:
performing a first processing on the plurality of RGB images to obtain first processed RGB images, wherein the first processing comprises: performing feature point matching on each two adjacent RGB images of the plurality of RGB images using a SURF algorithm;
performing a second processing on the first processed RGB images to obtain second processed RGB images, wherein the second processing comprises: confirming whether the feature point matching has been correctly performed on each two adjacent RGB images of the first processed RGB images, and eliminating wrongly matched feature point; and
performing a third processing on the second processed RGB images to obtain third processed RGB images, wherein the third processing comprises: calculating a graphic angle difference of each two adjacent RGB images of the second processed RGB images using a RANSAC algorithm, and making the graphic angle of each two adjacent RGB images the same, by correspondingly correcting one of the each two adjacent RGB images based on the graphic angle difference.

### Claim 15 (depends on 13)

The non-transitory storage medium as claimed in claim 13, wherein the constructing a three-dimensional map based on the plurality of fused images comprises:
calculating three-dimensional coordinates of each pixel point of each fused image of the plurality of fused images in a physical space;
associating each fused image with the three-dimensional coordinates of each pixel point of the each fused image, and stitching the plurality of fused images to obtain the three-dimensional map.

### Claim 16 (depends on 15)

The non-transitory storage medium as claimed in claim 15, wherein the three-dimensional coordinates of each pixel point p1 of the each fused image is (x1, y1, z1), z1=d/s; x1=(xx1−cx)*z1/fx; y1=(yy1−cy)*z1/fy; xx1 representing an abscissa of the pixel point p1 in the fused image, yy1 representing an ordinate of the pixel point p1 in the fused image, and d representing a vertical coordinate of the pixel point p1 in the fused image; fx representing a focal length of the RGB camera of depth camera module on an x-axis of the coordinate system, and fy representing the focal length of the RGB camera of depth camera module on a y-axis; and cx representing a distance from a center of an aperture of the RGB camera of depth camera module to the x-axis, cy representing a distance from the center of the aperture of the RGB camera of depth camera module to the y axis; and s representing a zoom value of the RGB camera of depth camera module.

### Claim 17 (depends on 16)

The non-transitory storage medium as claimed in claim 16, wherein the controlling the robotic arm to grasp objects and place objects based on the three-dimensional map comprises:
locating position coordinates of the robotic arm based on the three-dimensional map, when the three-dimensional map obtained;
obtaining first position coordinates of a target object, wherein the first position coordinates are coordinates of a current position of the target object;
controlling the robotic arm to grasp the target object based on the position coordinates of the robotic arm and the first position coordinates of the target object;
obtaining second position coordinates of the target object, wherein the second position coordinates are coordinates of a target position where the target object needs to be placed; and
controlling the robotic arm to place the target object on the target position based on the second position coordinates of the target object.

### Claim 18 (depends on 17)

The non-transitory storage medium as claimed in claim 17, wherein the controlling the robotic arm to grasp objects and place objects based on the three-dimensional map further comprises:
determining whether the robotic arm grasps the target object based on the position coordinates of the robotic arm and the first position coordinates of the target object;
recognizing the target object and measuring position coordinates of the target object when the robotic arm fails to grasp the target object; and
controlling the robotic arm to grasp the target object based on the measured position coordinates of the target object.
