# US 2021/0241491 A1 · Method and system for performing automatic camera calibration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241491 A1
- Document ID: US20210241491A1
- Status: Published application
- Kind: A1
- Application number: 16/871,361
- Filed: 2020-05-11
- Published: 2021-08-05
- Earliest claimed: 2020-02-04
- Assignees: MUJIN, Inc.
- Applicants: Mujin, Inc.
- Inventors: Russell ISLAM; Xutao YE
- Web page: https://patentgrep.com/patent/US20210241491A1

## CPC

- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- B25J 9/1697: {Vision controlled systems}
- G05B 2219/37009: Calibration of vision system, camera, adapt light level
- G05B 2219/39008: Fixed camera detects reference pattern held by end effector
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30208: Marker matrix
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A system and method for performing automatic camera calibration is provided. The system receives a calibration image, and determines a plurality of image coordinates for representing respective locations at which a plurality of pattern elements of a calibration pattern appear in a calibration image. The system determines, based on the plurality of image coordinates and defined pattern element coordinates, an estimate for a first lens distortion parameter of a set of lens distortion parameters, wherein the estimate for the first lens distortion parameter is determined while estimating a second lens distortion parameter of the set of lens distortion parameters to be zero, or is determined without estimating the second lens distortion parameter. The system determines, after the estimate of the first lens distortion parameter is determined, an estimate for the second lens distortion parameter based on the estimate for the first lens distortion parameter.

## Claims

### Claim 1

A computing system comprising:
a communication interface configured to communicate with a camera having a camera field of view;
a control circuit configured, when the camera has generated a calibration image for a calibration pattern in the camera field of view, and when the calibration pattern includes a plurality of pattern elements having respective defined pattern element coordinates in a pattern coordinate system, to perform camera calibration by:
receiving the calibration image, the calibration image being an image representing the calibration pattern;
determining a plurality of image coordinates for representing respective locations at which the plurality of pattern elements appear in the calibration image;
determining, based on the plurality of image coordinates and the defined pattern element coordinates, an estimate for a first lens distortion parameter of a set of lens distortion parameters which describe lens distortion associated with the camera, wherein the estimate for the first lens distortion parameter is determined while estimating a second lens distortion parameter of the set of lens distortion parameters to be zero, or is determined without estimating the second lens distortion parameter;
determining, after the estimate of the first lens distortion parameter is determined, an estimate for the second lens distortion parameter based on the estimate for the first lens distortion parameter; and
determining camera calibration information that includes respective estimates for the set of lens distortion parameters, wherein the respective estimates for the set of lens distortion parameters includes or are based on the estimate for the first lens distortion parameter and the estimate for the second lens distortion parameter,
wherein the control circuit is further configured, when the communication interface is in communication with the camera and with a robot, to receive a subsequent image generated by the camera after the camera calibration has been performed, and to generate a movement command for controlling robot movement, wherein the movement command is based on the subsequent image and based on the camera calibration information.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein the control circuit is configured to determine the estimate for the first lens distortion parameter while estimating all other lens distortion parameters of the set of lens distortion parameters to be zero, or is configured to determine the estimate for the first lens distortion parameter without estimating any other lens distortion parameter of the set of lens distortion parameters.

### Claim 3 (depends on 1)

The computing system of claim 1, wherein the first lens distortion parameter describes a first type of lens distortion associated with the camera, and the second lens distortion parameter describes a second type of lens distortion associated with the camera.

### Claim 4 (depends on 3)

The computing system of claim 3, wherein the first type of lens distortion is radial lens distortion, and the second type of lens distortion is tangential lens distortion.

### Claim 5 (depends on 1)

The computing system of claim 1, wherein the first lens distortion parameter and the second lens distortion parameter describes a same type of lens distortion associated with the camera.

### Claim 6 (depends on 5)

The computing system of claim 5, wherein the set of lens distortion parameters include multiple lens distortion parameters that describe multiple respective radial polynomial components which are part of a model of radial lens distortion associated with the camera, and wherein the first lens distortion parameter is one of the multiple lens distortion parameters and describes a lowest-degree radial polynomial component among the multiple respective radial polynomial components.

### Claim 7 (depends on 6)

The computing system of claim 6, wherein the set of lens distortion parameters include a third lens distortion parameter, wherein the third lens distortion parameter describes a highest-degree radial polynomial component among the multiple respective radial polynomial components, and
wherein the estimate for the second lens distortion parameter is determined based on the estimate for the first lens distortion parameter, and is determined while estimating the third lens distortion parameter to be zero, or without estimating the third lens distortion parameter.

### Claim 8 (depends on 7)

The computing system of claim 7, wherein the estimate for the first lens distortion parameter is a first estimate thereof, and is determined during a first camera calibration stage, and wherein the estimate for the second lens distortion parameter is determined during a subsequent camera calibration stage that follows the first camera calibration stage,
wherein the set of lens distortion parameters include the third lens distortion parameter and a subset having other lens distortion parameters of the set,
wherein the control circuit is configured, during the subsequent camera calibration stage, to estimate lens distortion parameters of the subset of lens distortion parameters while estimating the third lens distortion parameter to be zero, or without estimating the third lens distortion parameter,
wherein the estimating of lens distortion parameters of the subset includes determining the estimate of the second lens distortion parameter and determining a second estimate of the first lens distortion parameter.

### Claim 9 (depends on 1)

The computing system of claim 1, wherein the camera calibration information describes a set of projection parameters that describe camera image projection associated with the camera, wherein the control circuit is configured:
to determine respective estimates for the set of projection parameters;
to determine, after the estimate for the first lens distortion parameter and the estimate for the second lens distortion parameter are determined, an updated estimate for the first lens distortion parameter and an updated estimate for the second lens distortion parameter based on the respective estimates for the set of projection parameters, and
wherein the updated estimate for the first lens distortion parameter and the updated estimate for the second lens distortion parameter are determined while the set of projection parameters are fixed in value at the respective estimates thereof.

### Claim 10 (depends on 9)

The computing system of claim 9, wherein the updated estimate of the first lens distortion parameter and the updated estimate of the second lens distortion parameter are determined with a curvature reduction stage in which the control circuit is configured to perform the following:
(a) determining, for the curvature reduction stage, an initial estimate for the first lens distortion parameter and an initial estimate for the second lens distortion parameter based on the estimate of the first lens distortion parameter and based on the estimate of the second lens distortion parameter;
(b) generating, based on the initial estimate of the first lens distortion parameter, on the initial estimate of the second lens distortion parameter, and on the calibration image, a modified version of the calibration image that compensates against the lens distortion associated with the camera,
(c) determining an amount of curvature in the modified version of the calibration image,
(d) adjusting the initial estimate of the first lens distortion parameter and the initial estimate of the second lens distortion parameter based on the amount of curvature in the modified version of the calibration image so as to generate an adjusted estimate for the first lens distortion parameter and an adjusted estimate for the second lens distortion parameter that reduce the amount of curvature, wherein the adjusted estimate for the first lens distortion parameter is the updated estimate thereof, and the adjusted estimate for the second lens distortion parameter is the updated estimate thereof, and
wherein the control circuit is configured to determine the camera calibration information based on the updated estimate of the first lens distortion parameter and the updated estimate of the second lens distortion parameter.

### Claim 11 (depends on 9)

The computing system of claim 9, wherein the set of lens distortion parameters includes a third lens distortion parameter, wherein an estimate of the third lens distortion parameter is determined in the curvature reduction stage along with the updated estimate for the first lens distortion parameter and the updated estimate for the second lens distortion parameter,
wherein the control circuit is configured, after the updated estimate of the first lens distortion parameter and the updated estimate of the second lens distortion parameter have been determined, to determine respective updated estimates for the set of projection parameters based on the updated estimate of the first lens distortion parameter, the updated estimate of the second lens distortion parameter, and the estimate of the third lens distortion parameter while fixing the estimate of the third lens distortion parameter in value at the estimate thereof.

### Claim 12 (depends on 1)

The computing system of claim 1, wherein the set of lens distortion parameters include an additional lens distortion parameter other than the first lens distortion parameter and the second lens distortion parameter,
wherein the control circuit is configured to:
(a) determine an updated estimate for the first lens distortion parameter based on the estimate of the first lens distortion parameter and based on the estimate of the second lens distortion parameter, and
(b) determine an estimate for the additional lens distortion parameter based on the updated estimate for the first lens distortion parameter, wherein the estimate for the additional lens distortion parameter is determined while the first lens distortion parameter is fixed in value at the updated estimate thereof.

### Claim 13 (depends on 1)

The computing system of claim 1, wherein the camera with which the communication interface is configured to communicate is a first camera, wherein the calibration image is a first calibration image, and the camera field of view is a first camera field of view, and wherein the communication interface is further configured to communicate with a second camera having a second camera field of view,
wherein the control circuit is configured, when the calibration pattern is or has been in the first camera field of view and in the second camera field of view, to:
receive a second calibration image, wherein the second calibration image is a second image representing the calibration pattern and is generated by the second camera;
determine an estimate of a transformation function for describing a spatial relationship between the first camera and the second camera;
determine, based on the first calibration image, a first plurality of coordinates that describe respective locations of the plurality of the pattern elements relative to the first camera;
determine, based on the second calibration image, a second plurality of coordinates that describe respective locations of the plurality of the pattern elements relative to the second camera;
transform, based on the estimate of the transformation function, the second plurality of coordinates into a plurality of transformed coordinates that are relative to the first camera;
determine an error parameter value that describes respective differences between the first plurality of coordinates and the plurality of transformed coordinates; and
update the estimate of the transformation function based on the error parameter value to generate an updated estimate of the transformation function.

### Claim 14 (depends on 13)

The computing system of claim 13, wherein the first plurality of coordinates are the plurality of respective image coordinates that represent respective locations at which the plurality of pattern elements appear in the first calibration image, and
wherein the plurality of transformed coordinates are an additional plurality of respective image coordinates representing the plurality of pattern elements, and wherein the control circuit is configured to determine the additional plurality of respective image coordinates based on the camera calibration information for describing calibration for the first camera.

### Claim 15 (depends on 13)

The computing system of claim 13, wherein the first plurality of coordinates are a plurality of 3D coordinates for the pattern elements in a first camera coordinate system, which is a coordinate system for the first camera, and
wherein the plurality of transformed coordinates are an additional plurality of 3D coordinates for the pattern elements in the first camera coordinate system.

### Claim 16 (depends on 15)

The computing system of claim 15, wherein the error parameter value is based on values representing respective distances between the first plurality of 3D coordinates and the additional plurality of 3D coordinates.

### Claim 17 (depends on 15)

The computing system of claim 15, wherein the error parameter value is based on values representing respective angles formed between respective pairs of imaginary lines which extend from a location associated with the first camera, each of the respective pairs of imaginary lines have a respective first imaginary line extending to a respective 3D coordinate of the first plurality of 3D coordinates, and having a respective second imaginary line extending to a respective 3D coordinate of the additional plurality of 3D coordinates.

### Claim 18

A non-transitory computer-readable medium having instructions thereon that, when executed by a control circuit of a computing system, causes the control circuit to:
receive a calibration image, wherein the calibration image is received from the non-transitory computer-readable medium or via a communication interface of the computing system, the communication interface being configured to communicate with a camera having a camera field of view, wherein the calibration image is generated by the camera when a calibration pattern is in the camera field of view, wherein the calibration image is an image representing the calibration pattern, the calibration pattern having a plurality of pattern elements having respective defined pattern element coordinates in a pattern coordinate system;
determine a plurality of image coordinates that for representing respective locations at which the plurality of pattern elements appear in the calibration image;
determine, based on the plurality of image coordinates and the defined pattern element coordinates, an estimate for a first lens distortion parameter of a set of lens distortion parameters which describe lens distortion associated with the camera, wherein the estimate for the first lens distortion parameter is determined while estimating a second lens distortion parameter of the set of lens distortion parameters to be zero, or is determined without estimating the second lens distortion parameter;
determine, after the estimate of the first lens distortion parameter is determined, an estimate for the second lens distortion parameter based on the estimate of the first lens distortion parameter; and
determine camera calibration information that includes respective estimates for the set of lens distortion parameters, wherein the respective estimates for the set of lens distortion parameters includes or are based on the estimate for the first lens distortion parameter and the estimate for the second lens distortion parameter,
wherein the instructions, when executed by the control circuit and when the communication interface is in communication with the camera and with a robot, further cause the control circuit to receive a subsequent image generated by the camera after the camera calibration has been performed, and to generate a movement command for controlling robot movement, wherein the movement command is based on the subsequent image and based on the camera calibration information.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by the control circuit, further cause the control circuit to determine the estimate for the first lens distortion parameter while estimating all other lens distortion parameters of the set of lens distortion parameters to be zero, or to determine the estimate for the first lens distortion parameter without estimating any other lens distortion parameter of the set of lens distortion parameters.

### Claim 20

A method performed by a computing system for camera calibration, comprising:
receiving a calibration image by the computing system, wherein the computing system includes a communication interface configured to communicate with a camera having a camera field of view, wherein the calibration image is generated by the camera when a calibration pattern is in the camera field of view, wherein the calibration image is an image representing the calibration pattern, the calibration pattern having a plurality of pattern elements with respective defined pattern element coordinates in a pattern coordinate system;
determining a plurality of image coordinates for representing respective locations at which the plurality of pattern elements appear in the calibration image;
determining, based on the plurality of image coordinates and the defined pattern element coordinates, an estimate for a first lens distortion parameter of a set of lens distortion parameters which describe lens distortion associated with the camera, wherein the estimate for the first lens distortion parameter is determined while estimating a second lens distortion parameter of the set of lens distortion parameters to be zero, or is determined without estimating the second lens distortion parameter;
determining, after the estimate of the first lens distortion parameter is determined, an estimate for the second lens distortion parameter based on the estimate of the first lens distortion parameter;
determining camera calibration information that includes respective estimates for the set of lens distortion parameters, wherein the respective estimates for the set of lens distortion parameters includes or are based on the estimate for the first lens distortion parameter and the estimate for the second lens distortion parameter;
receiving a subsequent image generated by the camera after the camera calibration has been performed, and
generating a movement command for controlling robot movement, wherein the movement command is based on the subsequent image and based on the camera calibration information.