# US 2023/0419546 A1 · Online camera calibration for a mobile robot

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0419546 A1
- Document ID: US20230419546A1
- Status: Published application
- Kind: A1
- Application number: 18/335,661
- Filed: 2023-06-15
- Published: 2023-12-28
- Earliest claimed: 2022-06-23
- Assignees: Boston Dynamics, Inc.
- Applicants: Boston Dynamics, Inc.
- Inventors: Christopher Christopher; Dion Gonano
- Web page: https://patentgrep.com/patent/US20230419546A1

## CPC

- G06T 7/85: {Stereo camera calibration}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30244: Camera pose
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and apparatus for online camera calibration are provided. The method comprises receiving a first image captured by a first camera of a robot, wherein the first image includes an object having at least one known dimension, receiving a second image captured by a second camera of the robot, wherein the second image includes the object, wherein a field of view of the first camera and a field of view of the second camera at least partially overlap, projecting a plurality of points on the object in the first image to pixel locations in the second image, and determining, based on pixel locations of the plurality of points on the object in second image and the projected plurality of points on the object, a reprojection error.

## Claims

### Claim 1

A method, comprising:
receiving a first image captured by a first camera of a robot, wherein the first image includes an object having at least one known dimension;
receiving a second image captured by a second camera of the robot, wherein the second image includes the object, wherein a field of view of the first camera and a field of view of the second camera at least partially overlap;
projecting a plurality of points on the object in the first image to pixel locations in the second image; and
determining, based on pixel locations of the plurality of points on the object in second image and the projected plurality of points on the object, a reprojection error.

### Claim 2 (depends on 1)

The method of claim 1, wherein the object includes a plurality of corner points, and wherein the plurality of points on the object projected to pixel locations in the second image includes at least two of the plurality of corner points.

### Claim 3 (depends on 2)

The method of claim 2, wherein the object is a rectangle having four corner points, and wherein the plurality of points on the object projected to pixel locations in the second image includes the four corner points of the rectangle.

### Claim 4 (depends on 1)

The method of claim 1, wherein the object is a fiducial marker in an environment of the robot.

### Claim 5 (depends on 4)

The method of claim 4, wherein the fiducial marker is an AprilTag.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining the reprojection error comprises:
calculating, for each of the plurality of points on the object, a first distance between the point on the object in the second image and the pixel location of the corresponding projected point in the second image; and
determining the reprojection error based on the calculated first distances.

### Claim 7 (depends on 6)

The method of claim 6, wherein determining the reprojection error based on the calculated distances comprises:
calculating a second distance of a longest edge of the object along two of the plurality of points on the object;
dividing each of the calculated first distances by the second distance to generate normalized first distances; and
determining the reprojection error as an average of the normalized first distances.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first camera is a vision camera and the second camera is a depth camera.

### Claim 9 (depends on 8)

The method of claim 8, wherein the depth camera is a stereo vision camera.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
generating an instruction to perform an action when the reprojection error is greater than a threshold value.

### Claim 11 (depends on 10)

The method of claim 10, wherein generating an instruction to perform an action when the reprojection error is greater than a threshold value comprises generating an alert.

### Claim 12 (depends on 10)

The method of claim 10, wherein generating an instruction to perform an action when the reprojection error is greater than a threshold value comprises generating an instruction to stop autonomous navigation of the robot.

### Claim 13 (depends on 10)

The method of claim 10, wherein generating an instruction to perform an action comprises generating an instruction to calibrate one or more parameters associated with the first camera and/or the second camera based on the reprojection error.

### Claim 14 (depends on 13)

The method of claim 13, wherein calibrating one or more parameters associated with the first camera and/or the second camera comprises updating a lens model for the first camera and/or the second camera.

### Claim 15 (depends on 13)

The method of claim 13, wherein
the robot is configured to use an extrinsics transform to relate a first coordinate system of the first camera to a second coordinate system of the second camera, and
calibrating one or more parameters associated with the first camera and/or the second camera comprises updating the extrinsics transform.

### Claim 16 (depends on 15)

The method of claim 15, wherein updating the extrinsics transform comprises:
capturing a set of first images from the first camera, wherein each of the first images in the set includes the object;
capturing a set of second images from the second camera, wherein each of the second images in the set includes the object, each of the first images having a corresponding second image in the set of second image taken at a same time as the first image using a same pose;
performing a non-linear optimization over the first set of images and the second set of images to minimize the reprojection error for pairs of images from the first set and the second set, wherein an output of the non-linear optimization is a current extrinsics transform; and
updating the extrinsics transform used by the robot based on the current extrinsics transform output from the non-linear optimization.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
determining a pose of the robot using the updated extrinsics transform.

### Claim 18

A robot, comprising:
a perception system including:
a first camera configured to capture a first image, wherein the first image includes an object having at least one known dimension; and
a second camera configured to capture a second image, wherein the second image includes the object, wherein a field of view of the first camera and a field of view of the second camera at least partially overlap; and
at least one computer processor configured to:
project a plurality of points on the object in the first image to pixel locations in the second image; and
determine, based on pixel locations of the plurality of points on the object in second image and the projected plurality of points on the object, a reprojection error.

### Claim 19 (depends on 18)

The robot of claim 18, wherein the object includes a plurality of corner points, and wherein the plurality of points on the object projected to pixel locations in the second image includes at least two of the plurality of corner points.

### Claim at position 20

20-35. (canceled)

### Claim 36

A non-transitory computer readable medium encoded with a plurality of instructions that, when executed by at least one computer processor perform a method, the method comprising:
receiving a first image captured by a first camera of a robot, wherein the first image includes an object having at least one known dimension;
receiving a second image captured by a second camera of the robot, wherein the second image includes the object, wherein a field of view of the first camera and a field of view of the second camera at least partially overlap;
projecting a plurality of points on the object in the first image to pixel locations in the second image; and
determining, based on pixel locations of the plurality of points on the object in second image and the projected plurality of points on the object, a reprojection error.

### Claim at position 22

37-52. (canceled)
