# US 10,864,639 B2 · Simultaneous kinematic and hand-eye calibration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,864,639 B2
- Document ID: US10864639B2
- Status: Granted
- Kind: B2
- Application number: 16/448,925
- Filed: 2019-06-21
- Published: 2019-10-10
- Granted: 2020-12-15
- Earliest claimed: 2016-09-28
- Assignees: Cognex Corporation
- Applicants: Cognex Corporation
- Inventors: Lifeng Liu; Cyril C. Marrion; Tian Gan; David Michael; Han Xiao
- Web page: https://patentgrep.com/patent/US10864639B2

## CPC

- B25J 9/1697: {Vision controlled systems}
- B25J 9/1692: {Calibration of manipulator}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06T 7/74: {involving reference images or patches}
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G05B 2219/39016: Simultaneous calibration of manipulator and camera
- G05B 2219/39045: Camera on end effector detects reference pattern
- G05B 2219/39057: Hand eye calibration, eye, camera on hand, end effector
- G05B 2219/40607: Fixed camera to observe workspace, object, workpiece, global
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Described are machine vision systems and methods for simultaneous kinematic and hand-eye calibration. A machine vision system includes a robot and a 3D sensor in communication with a control system. The control system is configured to move the robot to poses, and for each pose: capture a 3D image of calibration target features and robot joint angles. The control system is configured to obtain initial values for robot calibration parameters, and determine initial values for hand-eye calibration parameters based on the initial values for the robot calibration parameters, the 3D image, and joint angles. The control system is configured to determine final values for the hand-eye calibration parameters and robot calibration parameters by refining the hand-eye calibration parameters and robot calibration parameters to minimize a cost function.

## Claims

### Claim 1

A machine vision system comprising:
a robot;
a 3D sensor; and
a control system in communication with the robot and the 3D sensor, the control system configured to:
move the robot to a plurality of poses, and for each pose of the plurality of poses:
capture, with the 3D sensor, a 3D image of one or more features on a calibration target; and
capture a plurality of robot joint angles;
obtain a plurality of initial values for a plurality of robot calibration parameters;
determine a plurality of initial values for a plurality of hand-eye calibration parameters based on the plurality of initial values for the plurality of robot calibration parameters, and the 3D image and the plurality of robot joint angles captured for each of the plurality of poses;
determine a plurality of final values for the plurality of hand-eye calibration parameters and a plurality of final values for the robot calibration parameters by refining one or more of the plurality of hand-eye calibration parameters and one or more of the plurality of robot calibration parameters to minimize a cost function.

### Claim 2 (depends on 1)

The machine vision system of claim 1, wherein the control system is further configured to:
use the plurality of final values for the plurality of hand-eye calibration parameters and the plurality of final values for the robot calibration parameters for a vision guided robot application.

### Claim 3 (depends on 1)

The machine vision system of claim 1, wherein the control system is further configured to:
move the robot using the plurality of final values for the robot calibration parameters.

### Claim 4 (depends on 1)

The machine vision system of claim 1, wherein the cost function measures, for each pose of the plurality of poses, a difference between an observed location of a feature in the 3D image captured for the pose and an expected location of the feature, the expected location determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of robot joint angles captured for the pose.

### Claim 5 (depends on 1)

The machine vision system of claim 1, wherein the cost function measures, for each pose of the plurality of poses, a difference between a composite transformation and an identity transformation, the composite transformation being a composition of a first transformation from a first coordinate space to a second coordinate space and a second transformation from the second coordinate space to the first coordinate space, the first transformation and the second transformation determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, the plurality of observed features of the calibration target for the pose, and the plurality of robot joint angles captured for the pose.

### Claim 6 (depends on 1)

The machine vision system of claim 1, wherein the cost function measures, for each pose of the plurality of poses, a difference between a robot joint angle of the plurality of robot joint angles captured for the pose and an estimated joint angle for the pose, the estimated joint angle determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of observed features of the calibration target for the pose.

### Claim 7 (depends on 1)

The machine vision system of claim 1, wherein the control system is further configured to minimize the cost function using non-linear least squares.

### Claim 8 (depends on 1)

The machine vision system of claim 1, wherein the control system is configured to refine less than all of the plurality of hand-eye calibration parameters and less than all of the plurality of robot calibration parameters.

### Claim 9 (depends on 1)

The machine vision system of claim 1, wherein the plurality of robot calibration parameters comprises Denavit-Hartenberg parameters for the robot.

### Claim 10

A machine vision system-implemented calibration method comprising:
moving, by a control system, a robot to a plurality of poses, and for each pose of the plurality of poses:
capturing, by a 3D sensor, a 3D image of one or more features on a calibration target; and
capturing, by the control system, a plurality of robot joint angles;
obtaining, by the control system, a plurality of initial values for a plurality of robot calibration parameters;
determining, by the control system, a plurality of initial values for a plurality of hand-eye calibration parameters based on the plurality of initial values for the plurality of robot calibration parameters, and the 3D image and the plurality of robot joint angles captured for each of the plurality of poses;
determining, by the control system, a plurality of final values for the plurality of hand-eye calibration parameters and a plurality of final values for the plurality of robot calibration parameters by refining one or more of the plurality of hand-eye calibration parameters and one or more of the plurality of robot calibration parameters to minimize a cost function.

### Claim 11 (depends on 10)

The method of claim 10, wherein the cost function measures, for each pose of the plurality of poses, a difference between an observed location of a feature in the 3D image captured for the pose and an expected location of the feature, the expected location determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of robot joint angles captured for the pose.

### Claim 12 (depends on 10)

The method of claim 10, wherein the cost function measures, for each pose of the plurality of poses, a difference between a composite transformation and an identity transformation, the composite transformation being a composition of a first transformation from a first coordinate space to a second coordinate space and a second transformation from the second coordinate space to the first coordinate space, the first transformation and the second transformation determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, the plurality of observed features of the calibration target for the pose, and the plurality of robot joint angles captured for the pose.

### Claim 13 (depends on 10)

The method of claim 10, wherein the cost function measures, for each pose of the plurality of poses, a difference between a robot joint angle of the plurality of robot joint angles captured for the pose and an estimated joint angle for the pose, the estimated joint angle determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of observed features of the calibration target for the pose.

### Claim 14 (depends on 10)

The method of claim 10, further comprising minimizing the cost function using non-linear least squares.

### Claim 15 (depends on 10)

The method of claim 10, further comprising refining less than all of the plurality of hand-eye calibration parameters and less than all of the plurality of robot calibration parameters.

### Claim 16 (depends on 10)

The method of claim 10, wherein the plurality of robot calibration parameters comprises Denavit-Hartenberg parameters for the robot.