# US 12,759,159 B2 · Robot system and control apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,759,159 B2
- Document ID: US12759159B2
- Status: Granted
- Kind: B2
- Application number: 18/273,817
- Filed: 2022-02-14
- Published: 2024-09-19
- Granted: 2026-10-13
- Earliest claimed: 2021-02-18
- Assignees: Fanuc Corporation
- Applicants: Fanuc Corporation
- Inventors: Toshiyuki Ando
- Web page: https://patentgrep.com/patent/US12759159B2

## CPC

- B25J 9/1697: {Vision controlled systems}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})

## Abstract

A robot system according to one aspect of the present disclosure includes a robot arm mechanism to which a hand for gripping a workpiece stored in a container with an opening at a top of the container is attached, a sensor configured to obtain two-dimensional image data including the container and three-dimensional point cloud data including the container, and a control apparatus configured to identify a position, direction, and size of the opening of the container based on the image data and the point cloud data and control the robot arm mechanism so as not to interfere with the container. It is possible to identify the position, orientation, and size of the opening of the container without performing a touch-up operation on the container.

## Claims

### Claim 1

A robot system for taking out a workpiece stored in a container with an opening at a top of the container, wherein a plurality of markers are disposed at a plurality of locations of the container, the robot system comprising:
a robot arm mechanism to which a hand for gripping the workpiece is attached;
a sensor configured to generate image data and point cloud data of the container, wherein
the image data contain color information related to the container,
the point cloud data include three-dimensional coordinate information related to a surface of the container,
the image data are expressed in two-dimensional coordinates (X, Y), and
the point cloud data are expressed in three-dimensional coordinates (X, Y, Z),
two-dimensional coordinate components (X, Y) of the three-dimensional coordinates (X, Y, Z) in the point cloud data are associated with the two-dimensional coordinates (X, Y) of the image data based on a correspondence relationship between the image data and the point cloud data; and
a control apparatus configured to:
extract the plurality of markers as feature points from the image data by image processing,
identify, from the point cloud data, three-dimensional coordinates (X, Y, Z) corresponding to two-dimensional coordinates (X, Y) of the plurality of markers extracted from the image data by referencing the correspondence relationship between the image data and the point cloud data without projecting the point cloud data onto the image data, thereby converting the two-dimensional coordinates (X, Y) of the plurality of markers into three-dimensional coordinates (X, Y, Z) of the plurality of markers,
identify a position, orientation, and size of the opening of the container based on the three-dimensional coordinates (X, Y, Z) of the plurality of markers, and
control the robot arm mechanism so as not to interfere with the container based on the identified position, orientation, and size of the opening of the container.

### Claim 2 (depends on 1)

The robot system according to claim 1, wherein
the opening has a rectangular shape, and
the control apparatus is configured to:
extract, as the plurality of markers, three feature points provided on upper end surfaces of three corners of the container from the image data;
identify, from the point cloud data, three-dimensional coordinates (X, Y, Z) corresponding to two-dimensional coordinates (X, Y) of the extracted three feature points by referencing the correspondence relationship between the image data and the point cloud data without projecting the point cloud data onto the image data; and
identify the position, orientation, and size of the opening of the container based on the identified three-dimensional coordinates.

### Claim 3 (depends on 1)

The robot system according to claim 1, wherein
the control apparatus is configured to:
calculate a three-dimensional position of a center point of the container based on three-dimensional positions of three markers of the plurality of markers,
scan the point cloud data from the three-dimensional positions of the three markers toward the three-dimensional position of the center point to identify three-dimensional positions of three corners of the opening of the container, and
identify the position, orientation, and size of the opening of the container based on the three-dimensional positions of the three corners of the opening of the container.

### Claim 4

A control apparatus for controlling a robot arm mechanism, the control apparatus comprising:
a reception unit configured to receive image data and point cloud data of a container;
a feature point extraction unit configured to extract a feature point from the image data;
a feature point position identification unit configured to identify, from the point cloud data, a three-dimensional position corresponding to a two-dimensional coordinate of the extracted feature point by referencing a correspondence relationship between the image data and the point cloud data without projecting the point cloud data onto the image data;
a container opening identification unit configured to identify a position, orientation, and size of an opening of the container based on the identified three-dimensional position; and
a robot control unit configured to control the robot arm mechanism based on the identified position, orientation, and size of the opening of the container.

### Claim 5 (depends on 1)

The robot system according to claim 1, wherein
the opening has a rectangular shape, and
the control apparatus includes a storage unit configured to store container information relating to a width direction and a length direction of the container; and
the control apparatus is configured to:
extract, from the image data, a feature point having a pattern enabling identification of the width direction and the length direction of the container;
identify, from the point cloud data, a three-dimensional position of the extracted feature point and directions of the container by referencing the correspondence relationship between the image data and the point cloud data without projecting the point cloud data onto the image data;
calculate a three-dimensional position of a center point of the container based on the identified feature point and the stored container information;
identify a three-dimensional position of a corner of the opening of the container based on the center point; and
identify the position, orientation, and size of the opening of the container.