# US 2024/0253236 A1 · Method for controlling robotic arm, electronic device, and computer-readable storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0253236 A1
- Document ID: US20240253236A1
- Status: Published application
- Kind: A1
- Application number: 18/566,332
- Filed: 2022-05-12
- Published: 2024-08-01
- Earliest claimed: 2021-06-04
- Assignees: Jingdong Technology Information Technology Co., Ltd.
- Applicants: Jingdong Technology Information Technology Co., Ltd.
- Inventors: David T; Chao XUE
- Web page: https://patentgrep.com/patent/US20240253236A1

## CPC

- B25J 9/1697: {Vision controlled systems}
- B25J 9/16: Program controls (total factory control, i.e. centrally controlling a plurality of machines, G05B19/418)
- B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for controlling a robotic arm includes: acquiring a first depth image and a first color image of a target object; predicting first predicted values and second predicted values of respective candidate actions of the robotic arm based on the first depth image and the first color image, in which the first predicted value represents a probability of the robotic arm separating the target object from an adjacent object by executing the corresponding candidate action; and the second predicted value represents a probability of the robotic arm successfully grabbing the target object by executing the corresponding candidate action; determining a target action based on the first predicted values and the second predicted values of the candidate actions; and controlling the robotic arm to execute the target action.

## Claims

### Claim 1

A method for controlling a robotic arm, comprising:
acquiring a first depth image and a first color image of a target object;
predicting first predicted values and second predicted values of respective candidate actions of the robotic arm based on the first depth image and the first color image, wherein the first predicted value represents a probability of the robotic arm separating the target object from an adjacent object by executing the corresponding candidate action; and the second predicted value represents a probability of the robotic arm successfully grabbing the target object by executing the corresponding candidate action;
determining a target action based on the first predicted values and the second predicted values of the candidate actions; and
controlling the robotic arm to execute the target action.

### Claim 2 (depends on 1)

The method according to claim 1, wherein predicting the first predicted values and the second predicted values of the candidate actions of the robotic arm based on the first depth image and the first color image comprises:
obtaining a plurality of input depth images by rotating the first depth image by a plurality of preset angles respectively along a preset rotation direction;
obtaining a plurality of input color images by rotating the first color image by the plurality of preset angles respectively along the preset rotation direction;
predicting the first predicted values of the candidate actions by inputting the plurality of input depth images and the plurality of input color images into a first prediction network; and
predicting the second predicted values of the candidate actions by inputting the plurality of input depth images and the plurality of input color images into a second prediction network.

### Claim 3 (depends on 1)

The method according to claim 1, further comprising:
acquiring a second depth image of the target object after executing the target action;
determining a first reward value of the target action by using a classification network based on the second depth image and the first depth image, wherein the first reward value is configured to indicate an effective degree of the robotic arm separating the target object from the adjacent object by executing the target action;
determining a second reward value of the target action based on whether the robotic arm successfully grabs the target object; and
training the first prediction network based on the first reward value, and training the second prediction network based on the second reward value.

### Claim 4 (depends on 3)

The method according to claim 3, wherein determining the first reward value of the target action by using the classification network based on the second depth image and the first depth image comprises:
extracting features of the first depth image and the second depth image respectively by using a feature extraction layer of the classification network;
obtaining a first fusion feature by fusing the feature of the first depth image with the feature of the second depth image; and
obtaining the first reward value of the target action by performing a classification prediction on the first fusion feature by using a classification layer of the classification network.

### Claim 5 (depends on 2)

The method according to claim 2, wherein predicting the first predicted values of the candidate actions by inputting the plurality of input depth images and the plurality of input color images into the first prediction network comprises:
extracting features of the plurality of input depth images and the plurality of input color images by using a feature extraction layer of the first prediction network, and obtaining a plurality of first fusion feature images by fusing the features of the plurality of input depth images with the features of the plurality of input color images respectively;
reversely rotating each of the first fusion feature images along the preset rotation direction; and
obtaining the first predicted values of the candidate actions by performing an action prediction on the first fusion feature images reversely rotated by using a prediction layer of the first prediction network.

### Claim 6 (depends on 2)

The method according to claim 2, wherein predicting the second predicted values of the candidate actions by inputting the plurality of input depth images and the plurality of input color images into the second prediction network comprises:
extracting features of the plurality of input depth images and the plurality of input color images by using a feature extraction layer of the second prediction network, and obtaining a plurality of second fusion feature images by fusing features of the plurality of input depth images with features of the plurality of input color images respectively;
reversely rotating each of the second fusion feature images along the preset rotation direction; and
obtaining the second reward values of the candidate actions by performing an action prediction on the second fusion feature images reversely rotated by using a prediction layer of the second prediction network.

### Claim 7 (depends on 1)

The method according to claim 1, wherein determining the target action based on the first predicted values and the second predicted values of the candidate actions comprises:
correcting the first predicted values of the candidate actions based on a contour of the target object indicated by the first depth image;
correcting the second predicted values of the candidate actions based on a central position of the target object indicated by the first depth image; and
selecting the target action from the candidate actions based on the corrected first predicted values and the corrected second predicted values.

### Claim 8 (depends on 1)

The method according to claim 1, wherein acquiring the first depth image and the first color image of the target object comprises:
acquiring a captured original depth image and a captured original color image captured;
obtaining three-dimensional coordinate positions corresponding to pixel points in the original depth image and the original color image under a coordinate system of the robotic arm by converting the original depth image and the original color image to the coordinate system of the robotic arm;
obtaining two-dimensional coordinate points corresponding to the three-dimensional coordinate positions respectively by projecting the three-dimensional coordinate positions to a preset two-dimensional plane;
generating the first depth image based on depths of the pixel points corresponding to the two-dimensional coordinate points in the original depth image; and
generating the first color image based on colors of the pixel points corresponding to the two-dimensional coordinate points in the original color image.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the preset two-dimensional plane is determined based on a minimum working distance along a preset direction under the coordinate system of the robotic arm.

### Claim 10

(canceled)

### Claim 11

A computer device, comprising a memory, a processor and a computer program stored on the memory and executable by the processor, when executing the computer program, the processor is configured to:
acquire a first depth image and a first color image of a target object;
predict first predicted values and second predicted values of respective candidate actions of the robotic arm based on the first depth image and the first color image, wherein the first predicted value represents a probability of the robotic arm separating the target object from an adjacent object by executing the corresponding candidate action; and the second predicted value represents a probability of the robotic arm successfully grabbing the target object by executing the corresponding candidate action;
determine a target action based on the first predicted values and the second predicted values of the candidate actions; and
control the robotic arm to execute the target action.

### Claim 12

A non-transitory computer-readable storage medium with a computer program stored thereon, wherein, when executed by a processor, the computer program is caused to implement;
acquiring a first depth image and a first color image of a target object;
predicting first predicted values and second predicted values of respective candidate actions of the robotic arm based on the first depth image and the first color image, wherein the first predicted value represents a probability of the robotic arm separating the target object from an adjacent object by executing the corresponding candidate action; and the second predicted value represents a probability of the robotic arm successfully grabbing the target object by executing the corresponding candidate action;
determining a target action based on the first predicted values and the second predicted values of the candidate actions; and
controlling the robotic arm to execute the target action.

### Claim 13

(canceled)

### Claim 14

(canceled)

### Claim 15 (depends on 11)

The computer device according to claim 11, wherein the processor is configured to:
obtain a plurality of input depth images by rotating the first depth image by a plurality of preset angles respectively along a preset rotation direction;
obtain a plurality of input color images by rotating the first color image by the plurality of preset angles respectively along the preset rotation direction;
predict the first predicted values of the candidate actions by inputting the plurality of input depth images and the plurality of input color images into a first prediction network; and
predict the second predicted values of the candidate actions by inputting the plurality of input depth images and the plurality of input color images into a second prediction network.

### Claim 16 (depends on 11)

The computer device according to claim 11, wherein the processor is configured to:
acquire a second depth image of the target object after executing the target action;
determine a first reward value of the target action by using a classification network based on the second depth image and the first depth image, wherein the first reward value is configured to indicate an effective degree of the robotic arm separating the target object from the adjacent object by executing the target action;
determine a second reward value of the target action based on whether the robotic arm successfully grabs the target object; and
train the first prediction network based on the first reward value, and train the second prediction network based on the second reward value.

### Claim 17 (depends on 16)

The computer device according to claim 16, wherein the processor is configured to:
extract features of the first depth image and the second depth image respectively by using a feature extraction layer of the classification network;
obtain a first fusion feature by fusing the feature of the first depth image with the feature of the second depth image; and
obtain the first reward value of the target action by performing a classification prediction on the first fusion feature by using a classification layer of the classification network.

### Claim 18 (depends on 15)

The computer device according to claim 15, wherein the processor is configured to:
extract features of the plurality of input depth images and the plurality of input color images by using a feature extraction layer of the first prediction network, and obtain a plurality of first fusion feature images by fusing the features of the plurality of input depth images with the features of the plurality of input color images respectively;
reversely rotate each of the first fusion feature images along the preset rotation direction; and
obtain the first predicted values of the candidate actions by performing an action prediction on the first fusion feature images reversely rotated by using a prediction layer of the first prediction network.

### Claim 19 (depends on 15)

The computer device according to claim 15, wherein the processor is configured to:
extract features of the plurality of input depth images and the plurality of input color images by using a feature extraction layer of the second prediction network, and obtain a plurality of second fusion feature images by fusing features of the plurality of input depth images with features of the plurality of input color images respectively;
reversely rotate each of the second fusion feature images along the preset rotation direction; and
obtain the second reward values of the candidate actions by performing an action prediction on the second fusion feature images reversely rotated by using a prediction layer of the second prediction network.

### Claim 20 (depends on 11)

The computer device according to claim 11, wherein the processor is configured to:
correct the first predicted values of the candidate actions based on a contour of the target object indicated by the first depth image;
correct the second predicted values of the candidate actions based on a central position of the target object indicated by the first depth image; and
select the target action from the candidate actions based on the corrected first predicted values and the corrected second predicted values.

### Claim 21 (depends on 11)

The computer device according to claim 11, wherein the processor is configured to:
acquire a captured original depth image and a captured original color image captured;
obtain three-dimensional coordinate positions corresponding to pixel points in the original depth image and the original color image under a coordinate system of the robotic arm by converting the original depth image and the original color image to the coordinate system of the robotic arm;
obtain two-dimensional coordinate points corresponding to the three-dimensional coordinate positions respectively by projecting the three-dimensional coordinate positions to a preset two-dimensional plane;
generate the first depth image based on depths of the pixel points corresponding to the two-dimensional coordinate points in the original depth image; and
generate the first color image based on colors of the pixel points corresponding to the two-dimensional coordinate points in the original color image.

### Claim 22 (depends on 21)

The computer device according to claim 21, wherein the preset two-dimensional plane is determined based on a minimum working distance along a preset direction under the coordinate system of the robotic arm.
