# US 2022/0314439 A1 · System and method for object shape identification and pose measurement using multi-fingered hand

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0314439 A1
- Document ID: US20220314439A1
- Status: Published application
- Kind: A1
- Application number: 17/617,992
- Filed: 2020-06-11
- Published: 2022-10-06
- Earliest claimed: 2019-06-11
- Assignees: Magna International Inc.
- Applicants: Magna International Inc.
- Inventors: Chinpei TANG; Alexander ZAK
- Web page: https://patentgrep.com/patent/US20220314439A1

## CPC

- B25J 9/1612: {characterised by the hand, wrist, grip control}
- B25J 9/1694: {characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion}
- B25J 13/084: {Tactile sensors (in general G01L5/16, G01L5/22)}
- B25J 13/089: {Determining the position of the robot with reference to its environment}
- G05B 2219/39518: Rolling contact between fingers, robot arms and object
- G05B 2219/40575: Camera combined with tactile sensors, for 3-D
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A system and method for identifying a shape and pose of an object is provided. The system includes a controller and a grasping device. The grasping device includes a plurality of fingers that are moveable relative to a base. The fingers include one or more tactile sensors attached thereto. The tactile sensors are configured to collect data, such as a 2D image or 3D point cloud, based on points of contact with the object when the object is grasped. The grasping device is configured to roll the object within the fingers and collect additional data. The controller may combine the data and determine the shape and pose of the object based on the combined data.

## Claims

### Claim 1

A system for determining a physical characteristic and pose of an object, the system comprising:
a controller and a processor with instructions stored thereon;
a grasping device in communication with the controller, the grasping device configured to grasp an object, wherein the grasping device includes a plurality of fingers attached to a base and the fingers are moveable relative to the base; and
one or more tactile sensors attached to the fingers, the tactile sensors configured to detect first data associated with a physical characteristic of the object when the object is grasped by the grasping device;
wherein the controller is configured to receive the data from the tactile sensors and to store the data;
wherein the fingers are configured to roll the object within the fingers while the object is being grasped, and the tactile sensors are configured to provide second data to the controller associated with additional physical characteristics of the object;
wherein the controller is configured to obtain the second data associated with additional physical characteristics while the fingers are grasping the object and rolling the object; and
wherein the controller is configured to combine the first data and the second data into combined data, and configured to determine a shape and pose of the object from the combined data.

### Claim 2 (depends on 1)

The system of claim 1, wherein the first data and the second data are in the form of 2D images.

### Claim 3 (depends on 1)

The system of claim 1, wherein the first data and the second data are in the form of 3D point clouds.

### Claim 4 (depends on 1)

The system of claim 1, wherein the first data and the second data are obtained without visualization techniques.

### Claim 5 (depends on 1)

The system of claim 1, wherein the shape of the object is determined contemporaneously with grasping and manipulation of the object.

### Claim 6 (depends on 1)

The system of claim 1, wherein the plurality of fingers are pivotally attached to the base of the grasping device.

### Claim 7 (depends on 6)

The system of claim 6, wherein the grasping device is configured to manipulate the fingers relative to known predetermined coordinates of the base.

### Claim 8 (depends on 6)

The system of claim 6, wherein the fingers each include a tip portion coupled to and moveable relative to a proximal portion, wherein the proximal portion is attached to the base.

### Claim 9 (depends on 8)

The system of claim 8, wherein the tactile sensors are attached to the tip portions.

### Claim 10 (depends on 1)

The system of claim 1, wherein the tactile sensors are vision-based tactile sensors having a camera configured to measure three-dimensional force, wherein the camera captures deformation of a sensor layer.

### Claim 11 (depends on 1)

The system of claim 1, wherein the tactile sensor is a resistance or capacitive changing sensor or a piezoelectric changing sensor.

### Claim 12

A method for identifying a shape and pose of an object, the method comprising:
grasping an object with a grasping device, wherein the grasping device includes a plurality of fingers having tactile sensors attached thereto;
detecting a first physical characteristic at the tactile sensors and collecting a first data corresponding to the first physical characteristic;
receiving, at a controller, the first data from tactile sensors;
manipulating the fingers of the grasping device and manipulating the object within the fingers of the grasping device;
detecting a second physical characteristic at the tactile sensors and collecting a second data corresponding to the second physical characteristic;
wherein the second physical characteristics and second data are obtained while the object is being grasped and manipulated by the fingers of the grasping device;
receiving, at the controller, the second data from the tactile sensors;
combining the first and second data into combined data at the controller; and
at the controller, identifying a shape and pose of the object based on the combined data.

### Claim 13 (depends on 12)

The method of claim 12, further comprising moving some of the fingers out of contact with the object repeatedly and detecting additional data from the tactile sensors at different positions on the object.

### Claim 14 (depends on 12)

The method of claim 12, further comprising rolling the object within the fingers to obtain the first and second data.

### Claim 15 (depends on 12)

The method of claim 12, further comprising manipulating the fingers relative to a base portion of the grasping device, the base portion having known predetermined coordinates.

### Claim 16 (depends on 12)

The method of claim 12, wherein the first and second data are obtained from vision-based tactile sensors, wherein the method includes obtaining an image of sensor layer deformation.

### Claim 17 (depends on 12)

The method of claim 12, wherein the first and second data are obtained from a resistance or capacitive changing sensor or a piezoelectric changing sensor.

### Claim 18 (depends on 12)

The method of claim 12, further comprising obtaining a 2d image or 3d point cloud from the tactile sensors.

### Claim 19 (depends on 12)

The method of claim 12, further comprising obtaining shape information of the object contemporaneously with the grasping of the object.

### Claim 20 (depends on 12)

The method of claim 12, further comprising automatically creating, at the controller, a model of the object based on the first and second data.
