# US 2022/0019762 A1 · Three-dimensional position and posture recognition device and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0019762 A1
- Document ID: US20220019762A1
- Status: Published application
- Kind: A1
- Application number: 17/295,148
- Filed: 2019-12-19
- Published: 2022-01-20
- Earliest claimed: 2018-12-21
- Assignees: Hitachi, Ltd.
- Applicants: Hitachi, Ltd.
- Inventors: Atsutake KOSUGE; Takashi OSHIMA; Yukinori AKAMINE; Keisuke YAMAMOTO
- Web page: https://patentgrep.com/patent/US20220019762A1

## CPC

- G06V 20/653: {by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces}
- B25J 9/1697: {Vision controlled systems}
- B25J 13/089: {Determining the position of the robot with reference to its environment}
- G01B 11/00: Measuring arrangements characterised by the use of optical techniques
- G01B 11/26: for measuring angles or tapers; for testing the alignment of axes
- G06T 7/75: {involving models}
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A three-dimensional position and posture recognition device speeds estimation of a position posture and a gripping coordinate posture of a gripping target product. The device includes: a sensor unit configured to measure a distance between an image of an object and the object; and a processing unit configured to calculate an object type included in the image, read model data of each object from the external memory, and create structured model data having a resolution set for each object from the model data, generate measurement point cloud data of a plurality of resolutions from information on a distance between an image of the object and the object, perform a K neighborhood point search using the structured model data and the measurement point cloud data, and perform three-dimensional position recognition processing of the object by rotation and translation estimation regarding a point obtained from the K neighborhood point search.

## Claims

### Claim 1

A three-dimensional position recognition device, comprising:
an external memory configured to store model data of each object;
a sensor unit configured to measure a distance between an image of an object and the object; and
a processing unit connected to the external memory and the sensor unit and configured to
calculate an object type included in the image based on information from the sensor unit,
read model data of each object from the external memory according to the object type, and create structured model data having a resolution set for each object from the model data,
generate measurement point cloud data of a plurality of resolutions from information on a distance between an image of the object and the object from the sensor unit,
perform a K neighborhood point search using the structured model data and the measurement point cloud data of each resolution of the measurement point cloud data of the plurality of resolutions, and
perform three-dimensional position recognition processing of the object by rotation and translation estimation regarding a point obtained from the K neighborhood point search.

### Claim 2 (depends on 1)

The three-dimensional position recognition device according to claim 1, wherein
the processing unit includes an internal memory storing the structured model data, and is configured to refer to the structured model data stored in the internal memory, and perform the K neighborhood point search with respect to the measurement point cloud data.

### Claim 3 (depends on 2)

The three-dimensional position recognition device according to claim 2, wherein
the processing unit is configured to receive an input of an allowable error for each object, and includes a parameter calculating unit configured to calculate a resolution set for each object based on the allowable error.

### Claim 4 (depends on 3)

The three-dimensional position recognition device according to claim 3, wherein
the processing of the K neighborhood point search and the rotation and translation estimation is performed by dedicated hardware.

### Claim 5 (depends on 4)

The three-dimensional position recognition device according to claim 4, wherein
the dedicated hardware includes an internal memory for structural data storing the structured model data and an internal memory for measurement data storing measurement point cloud data having a predetermined resolution.

### Claim 6 (depends on 5)

The three-dimensional position recognition device according to claim 5, further comprising:
a measurement point cloud data having the predetermined resolution that is stored in the internal memory for measurement data; and
a distance calculation module configured to set one measurement point cloud data as a query point, and calculate in parallel distances to a plurality of points of the structured model data stored in the internal memory for structural data relative to the query point.

### Claim 7

A three-dimensional position recognition device, comprising:
an external memory configured to store model data of each object;
a sensor unit configured to acquire distance information between a two-dimensional image of an object and the object;
an image recognition unit configured to output an object type and an object region coordinate based on the two-dimensional image of the object acquired by the sensor unit;
a parameter calculating unit configured to output resolution for each object based on the object type and the object region coordinate from the image recognition unit, and an allowable error for each object;
a data thinning unit configured to output, based on the resolution from the parameter calculating unit, thinned data from model data of the object read from the external memory according to the object type from the image recognition unit;
a data structuring unit configured to generate structured model data from the thinned data;
an internal memory configured to store the structured model data generated by the data structuring unit;
a point cloud generating unit configured to generate the measurement point cloud data from the two-dimensional image acquired from the sensor unit;
a resolution conversion unit configured to output the measurement point cloud data as measurement point cloud data of a plurality of resolutions after resolution conversion;
a K neighborhood point searching unit configured to perform K neighborhood point search using the structured model data stored in the internal memory and the predetermined measurement point cloud data from the resolution conversion unit;
a rotation and translation estimating unit configured to perform rotation and translation estimation using output of the K neighborhood point searching unit, and the structured model data stored in the internal memory;
a first loop that repeats processing of the K neighborhood point searching unit and the rotation and translation estimating unit until an error of the structured model data stored in the internal memory and the measurement point cloud data of the resolution conversion unit is a certain value or less; and
a second loop that repeats the processing of the K neighborhood point searching unit and the rotation and translation estimating unit with respect to measurement point cloud data of the plurality of the resolutions from the resolution conversion unit.

### Claim 8

A three-dimensional position recognition method of an object in a three-dimensional position recognition device, comprising steps of:
storing model data of each object by an external memory;
measuring a distance between an image of an object and the object by a sensor unit;
by a processing unit connected to the external memory and the sensor unit,
creating structured model data having a resolution set for each object from the model data of each object;
generating measurement point cloud data of a plurality of resolutions from information on a distance between an image of the object from the sensor unit and the object;
performing a K neighborhood point search using the structured model data of the resolution and the measurement point cloud data of each resolution of the measurement point cloud data of the plurality of resolutions; and
performing three-dimensional position recognition processing of the object by rotation and translation estimation based on corresponding point information obtained from the K neighborhood point search.

### Claim 9 (depends on 8)

The three-dimensional position recognition method according to claim 8, wherein
the processing unit includes an internal memory for storing the structured model data, and is configured to refer to the structured model data stored in the internal memory with respect to the measurement point cloud data, and perform the K neighborhood point search.

### Claim 10 (depends on 9)

The three-dimensional position recognition method according to claim 9, wherein
the processing unit is configured to receive an input of an allowable error for each object, and has a parameter calculating unit configured to calculate a resolution set for each object based on the allowable error.
