# US 12,240,129 B2 · Methods and systems of generating camera models for camera calibration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,240,129 B2
- Document ID: US12240129B2
- Status: Granted
- Kind: B2
- Application number: 17/897,012
- Filed: 2022-08-26
- Published: 2023-03-09
- Granted: 2025-03-04
- Earliest claimed: 2021-08-29
- Assignees: Quartus Engineering Incorporated
- Applicants: Quartus Engineering Incorporated
- Inventors: Jesse Melrose; Matthew Pearson; Jordan Liss; Julia Wetherill; Brian Dunne
- Web page: https://patentgrep.com/patent/US12240129B2

## CPC

- B25J 9/1692: {Calibration of manipulator}
- B25J 9/1697: {Vision controlled systems}
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Methods of generating a camera model are provided. A robotic assembly is used to move a calibration assembly relative to a camera assembly through a series of poses. The calibration assembly comprises a calibration target and markers. The camera assembly comprises a mount, a camera having a field of view, and markers. The predetermined series of poses, together, cause the calibration target to pass through the field of view of the camera. The camera is used to generate a respective image of the calibration target. A tracker is used, to determine respective locations in space of the markers. A transformation function is generated that maps onto a three-dimensional space the stored coordinates and determined locations in space of the markers and features of the calibration target. The transformation functions are used to generate a model of parameters of the camera. Systems are also provided.

## Claims

### Claim 1

A method of generating a camera model, the method comprising:
(a) using a robotic assembly to move a calibration assembly relative to a camera assembly, or to move the camera assembly relative to the calibration assembly, through a predetermined series of poses,
wherein the calibration assembly comprises:
a calibration target; and
first, second, and third calibration assembly markers coupled to the calibration target at respective positions,
wherein the camera assembly comprises:
a mount;
a camera coupled to the mount at a respective location and having a field of view (FOV); and
first, second, and third camera assembly markers coupled to the mount at respective positions, and
wherein the predetermined series of poses, together, cause the calibration target to pass through at least a portion of the FOV of the camera;
(b) using the camera, at each pose of the predetermined series of poses, to generate a respective image of the calibration target;
(c) using a tracker, at each pose of the predetermined series of poses, to determine respective locations in space of the first, second, and third calibration assembly markers and respective locations in space of the first, second, and third camera assembly markers;
(d) for each respective image, generating a transformation function that maps onto a three-dimensional object space (i) stored coordinates of the first, second, and third calibration assembly markers, (ii) stored coordinates of the first, second, and third camera assembly markers, (iii) the determined locations in space, for that image, of the first, second, and third calibration assembly markers, (iv) the determined locations in space, for that image, of the first, second, and third camera assembly markers, and (v) features of the calibration target within the respective image; and
(e) using the transformation functions for the respective images to generate a model of extrinsic parameters and intrinsic parameters of the camera.

### Claim 2 (depends on 1)

The method of claim 1, wherein the calibration assembly further comprises a fourth calibration assembly marker coupled to the calibration target.

### Claim 3 (depends on 2)

The method of claim 2, wherein operation (c) comprises:
using a tracker, at each pose of the predetermined series of poses, to determine respective locations in space of the first, second, third, and fourth calibration assembly markers.

### Claim 4 (depends on 1)

The method of claim 1, wherein the camera assembly further comprises a fourth camera assembly marker coupled to the mount.

### Claim 5 (depends on 4)

The method of claim 4, wherein operation (c) comprises:
using a tracker, at each pose of the predetermined series of poses, to determine respective locations in space of the first, second, third, and fourth camera assembly markers.

### Claim 6 (depends on 1)

The method of claim 1, wherein the calibration assembly markers respectively comprise spherically mounted retroreflectors (SMRs).

### Claim 7 (depends on 1)

The method of claim 1, wherein the camera assembly markers respectively comprise spherically mounted retroreflectors (SMRs).

### Claim 8 (depends on 1)

The method of claim 1, wherein the mount comprises a pin-diamond pin mount.

### Claim 9 (depends on 1)

The method of claim 1, wherein the tracker comprises a laser tracker.

### Claim 10 (depends on 1)

The method of claim 1, wherein the method further comprises:
(f) determining the coordinates of the first, second, and third camera assembly markers in the camera mount datum frame.

### Claim 11 (depends on 10)

The method of claim 10, wherein the operation (f) is performed using a coordinate measuring machine (CMM).

### Claim 12 (depends on 1)

The method of claim 1, wherein the calibration target comprises a rectilinear checkerboard chart.

### Claim 13 (depends on 1)

The method of claim 1, wherein the calibration target comprises a self-identifying binary code.

### Claim 14 (depends on 13)

The method of claim 13, wherein the self-identifying binary code comprises CALTag or ARTag.

### Claim 15 (depends on 1)

The method of claim 1, wherein the method further comprises:
(g) determining the locations of features of the calibration target relative to the first, second, and third calibration assembly markers.

### Claim 16 (depends on 15)

The method of claim 15, wherein operation (g) is performed using an optical measuring machine (OMM).

### Claim 17 (depends on 1)

The method of claim 1, wherein the predetermined series of poses, together, cause the calibration target to generate a superchart.

### Claim 18 (depends on 17)

The method of claim 17, wherein the superchart comprises a hemispherical shape.

### Claim 19 (depends on 17)

The method of claim 17, wherein the superchart comprises multiple layers.

### Claim 20 (depends on 1)

The method of claim 1, wherein the method further comprises:
(h) for each respective image, processing the image before operation (d).

### Claim 21 (depends on 20)

The method of claim 20, wherein the processing of the image comprises at least one of object detection, smoothing, edge enhancing, and morphological operations.

### Claim 22 (depends on 1)

The method of claim 1, wherein the method further comprises:
(i) repeating operations (a) through (d) with a different predetermined series of poses to generate an audit data set of extrinsic parameters and intrinsic parameters of the camera.

### Claim 23 (depends on 22)

The method of claim 22, wherein the method further comprises:
(j) using the determined locations in space of the first, second, and third calibration assembly markers from the audit data set; the determined locations in space of the first, second, and third camera assembly markers from the audit data set; and the camera model to determine the image space error and the object space error of the camera model.

### Claim 24

A system for generating a camera model, the system comprising:
(a) a calibration assembly comprising:
a calibration target; and
first, second, and third calibration assembly markers coupled to the calibration target at respective positions;
(b) a camera assembly comprising:
a mount;
a camera coupled to the mount at a respective location and having a field of view (FOV); and
first, second, and third camera assembly markers coupled to the mount at respective positions;
(c) a robotic assembly coupled to at least one of the calibration assembly and the camera assembly;
(d) a tracker; and
(e) a computer system coupled to the camera, the robotic assembly, and the tracker, the computer system comprising at least one processor and at least one non-volatile computer-readable medium,
the at least one non-volatile computer-readable medium storing coordinates of the first, second, and third calibration assembly markers relative to one another and relative to the calibration target;
the at least one non-volatile computer-readable medium storing coordinates of the first, second, and third camera assembly markers relative to one another and relative to the camera;
the at least one non-volatile computer-readable medium further storing instructions for causing the processor to perform operations comprising:
instructing the robotic assembly to move the calibration assembly relative to the camera assembly, or to move the camera assembly relative to the calibration assembly, through a predetermined series of poses that, together, cause the calibration target to pass through at least a portion of the FOV of the camera;
instructing the camera, at each pose of the predetermined series of poses, to generate a respective image of the calibration target;
instructing the tracker, at each pose of the predetermined series of poses, to determine respective locations in space of the first, second, and third calibration assembly markers and respective locations in space of the first, second, and third camera assembly markers;
for each respective image, generating a transformation function that maps onto a three-dimensional object space (i) the stored coordinates of the first, second, and third calibration assembly markers, (ii) the stored coordinates of the first, second, and third camera assembly markers, (iii) the determined respective locations in space, for that image, of the first, second, and third calibration assembly markers, (iv) the determined respective locations in space, for that image, of the first, second, and third camera assembly markers, and (v) features of the calibration target within the respective image; and
using the transformation functions for the respective images to generate a model of extrinsic parameters and intrinsic parameters of the camera.

### Claim 25 (depends on 24)

The system of claim 24, wherein the calibration assembly further comprises a fourth calibration assembly marker coupled to the calibration target.

### Claim 26 (depends on 25)

The system of claim 25, wherein the at least one non-volatile computer-readable medium stores the coordinates of the fourth calibration assembly marker relative to the first, second, and third calibration assembly markers and relative to the calibration target.

### Claim 27 (depends on 25)

The system of claim 25, wherein:
the at least one non-volatile computer-readable medium stores coordinates of the fourth calibration assembly marker; and
the instructions further comprise:
instructing the tracker, at each pose of the predetermined series of poses, to determine the respective location in space of the fourth calibration assembly marker, and
for each respective image, generating a transformation function that maps onto a three-dimensional object space the stored coordinates of the fourth calibration assembly marker.

### Claim 28 (depends on 24)

The system of claim 24, wherein the camera assembly further comprises a fourth camera assembly marker coupled to the mount.

### Claim 29 (depends on 28)

The system of claim 28, wherein the at least one non-volatile computer-readable medium stores the coordinates of the fourth camera assembly marker relative to the first, second, and third camera assembly markers and relative to the camera.

### Claim 30 (depends on 29)

The system of claim 29, wherein:
the at least one non-volatile computer-readable medium stores coordinates of the fourth camera assembly marker; and
the instructions further comprise:
instructing the tracker, at each pose of the predetermined series of poses, to determine the respective location in space of the fourth camera assembly marker,
for each respective image, generating a transformation function that maps onto a three-dimensional object space the stored coordinates of the fourth camera assembly marker, and
for each respective image, generating a transformation function that maps onto a three-dimensional object space the determined coordinates of the fourth camera assembly marker.

### Claim 31 (depends on 24)

The system of claim 24, wherein the calibration assembly markers respectively comprise spherically mounted retroreflectors (SMRs).

### Claim 32 (depends on 24)

The system of claim 24, wherein the camera assembly markers respectively comprise spherically mounted retroreflectors (SMRs).

### Claim 33 (depends on 24)

The system of claim 24, wherein the mount comprises a pin-diamond pin mount.

### Claim 34 (depends on 24)

The system of claim 24, wherein the tracker comprises a laser tracker.

### Claim 35 (depends on 24)

The system of claim 24, wherein the stored coordinates of the first, second, and third camera assembly markers relative to one another and relative to the camera are determined using a coordinate measuring machine (CMM).

### Claim 36 (depends on 24)

The system of claim 24, wherein the calibration target comprises a rectilinear checkerboard chart.

### Claim 37 (depends on 24)

The system of claim 24, wherein the calibration target comprises a self-identifying binary code.

### Claim 38 (depends on 37)

The system of claim 37, wherein the self-identifying binary code comprises CALTag or ARTag.

### Claim 39 (depends on 24)

The system of claim 24, wherein the at least one non-volatile computer-readable medium stores coordinates of features of the calibration target relative to the first, second, and third calibration assembly markers.

### Claim 40 (depends on 39)

The system of claim 39, wherein the stored coordinates of features of the calibration target relative to the first, second, and third calibration assembly markers are determined using an optical measurement machine (OMM).

### Claim 41 (depends on 24)

The system of claim 24, wherein the predetermined series of poses, together, cause the calibration target to generate a superchart.

### Claim 42 (depends on 41)

The system of claim 41, wherein the superchart comprises a hemispherical shape.

### Claim 43 (depends on 41)

The system of claim 41, wherein the superchart comprises multiple layers.

### Claim 44 (depends on 24)

The system of claim 24, wherein the instructions further comprise:
for each respective image, processing the image before generating a transformation function.

### Claim 45 (depends on 44)

The system of claim 44, wherein the processing of the image comprises at least one of object detection, smoothing, edge enhancing, and morphological operations.

### Claim 46 (depends on 24)

The system of claim 24, wherein the instructions further comprise repeating the operations in the instructions with a different predetermined series of poses to generate an audit data set.

### Claim 47 (depends on 46)

The system of claim 46, wherein the instructions further comprise using the determined locations in space of the first, second, and third calibration assembly markers from the audit data set; the determined locations in space of the first, second, and third camera assembly markers from the audit data set; and the camera model to determine the image space error and the object space error of the camera model.