# US 2023/0410365 A1 · Maintaining calibration of an ibis camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410365 A1
- Document ID: US20230410365A1
- Status: Published application
- Kind: A1
- Application number: 17/807,259
- Filed: 2022-06-16
- Published: 2023-12-21
- Earliest claimed: 2022-06-16
- Assignees: Motional AD LLC
- Applicants: Motional AD LLC
- Inventors: Peter Leeman; Nijumudheen Muhassin; Jayesh Dwivedi
- Web page: https://patentgrep.com/patent/US20230410365A1

## CPC

- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- H04N 17/002: {for television cameras}
- H04N 23/6812: {based on additional sensors, e.g. acceleration sensors}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10044: Radar image
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Provided are methods for maintaining calibration of an IBIS camera, which can include receiving a first initial extrinsic calibration of a first camera having a first sensor; determining a first translation of the first sensor based on a second position thereof when the first camera captures a first image; determining a first extrinsic translation shift based at least upon the first translation; and updating the first initial extrinsic calibration to yield a first modified extrinsic calibration for the first camera. Systems and computer program products are also provided.

## Claims

### Claim 1

A method of extrinsic camera calibration, comprising:
receiving, using at least one processor, a first initial extrinsic calibration of a first camera having a first sensor, wherein the first initial extrinsic calibration includes a rotation matrix and an initial translation matrix and is based on a first position of the first sensor when the first camera captures an initial image;
determining, using the at least one processor, a first translation of the first sensor based on a second position thereof when the first camera captures a first image, the second position being different than the first position;
determining, using the at least one processor, a first extrinsic translation shift based at least upon the first translation, wherein the first extrinsic translation shift represents a first net translation between the first position and the second position of the first sensor; and
updating, using the at least one processor, the first initial extrinsic calibration to yield a first modified extrinsic calibration for the first camera based on the first extrinsic translation shift, wherein the first modified extrinsic calibration includes the rotation matrix and a first adjusted translation matrix given by a summation of the first extrinsic translation shift with the initial translation matrix.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining, using the at least one processor, a second translation of the first sensor based on a third position thereof when the first camera captures a second image, the third position being different than the first position;
determining, using the at least one processor, a second extrinsic translation shift based at least upon the second translation, wherein the second extrinsic translation shift represents a net translation between the first position and the third position of the first sensor; and
updating, using the at least one processor, the first initial extrinsic calibration to yield a second modified extrinsic calibration for the first camera based on the second extrinsic translation shift, wherein the second modified extrinsic calibration includes the rotation matrix and a second adjusted translation matrix given by a summation of the second extrinsic translation shift with the initial translation matrix.

### Claim 3 (depends on 1)

The method of claim 1, comprising:
receiving, using the at least one processor, a second initial extrinsic calibration of a second camera having a second sensor, wherein the second initial extrinsic calibration includes a rotation matrix and an initial translation matrix and is based on a first position of the second sensor when the second camera captures an initial image;
determining, using the at least one processor, a second translation of the second sensor based on the second position thereof when the second camera captures a first image, the second position being different than the first position;
determining, using the at least one processor, a second extrinsic translation shift based at least upon the second translation, wherein the second extrinsic translation shift represents a first net translation between the first position and the second position of the second sensor; and
updating, using the at least one processor, the second initial extrinsic calibration to yield a second modified extrinsic calibration for the second camera based on the second extrinsic translation shift, wherein the second modified extrinsic calibration includes the rotation matrix and a second adjusted translation matrix given by the summation of the second extrinsic translation shift with the initial translation matrix.

### Claim 4 (depends on 3)

The method of claim 3, further comprising fusing the data of the first modified extrinsic calibration with data from a third sensor to rectify the first image.

### Claim 5 (depends on 4)

The method of claim 4, wherein the third sensor comprises a LIDAR sensor or a radar sensor.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first camera comprises an IBIS camera.

### Claim 7 (depends on 3)

The method of claim 3, wherein the second camera comprises an IBIS camera.

### Claim 8 (depends on 3)

The method of claim 3, wherein the second sensor is stationary relative to the first sensor.

### Claim 9 (depends on 1)

The method of claim 1, comprising causing a vehicle to operate based at least in part on output of the first camera.

### Claim 10

A vehicle comprising:
at least one computer-readable medium storing computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions, the execution carrying out operations comprising:
receiving, using at least one processor, a first initial extrinsic calibration of a first camera having a first sensor, wherein the first initial extrinsic calibration includes a rotation matrix and an initial translation matrix and is based on a first position of the first sensor when the first camera captures an initial image;
determining, using the at least one processor, a first translation of the first sensor based on a second position thereof when the first camera captures a first image, the second position being different than the first position;
determining, using the at least one processor, a first extrinsic translation shift based at least upon the first translation, wherein the first extrinsic translation shift represents a first net translation between the first position and the second position of the first sensor; and
updating, using the at least one processor, the first initial extrinsic calibration to yield a first modified extrinsic calibration for the first camera based on the first extrinsic translation shift, wherein the first modified extrinsic calibration includes the rotation matrix and a first adjusted translation matrix given by a summation of the first extrinsic translation shift with the initial translation matrix.

### Claim 11

A non-transitory computer-readable storage medium comprising at least one program for execution using at least one processor of a first device, the at least one program including instructions which, when executed using the at least one processor, cause the first device to perform operations comprising:
receiving, using the at least one processor, a first initial extrinsic calibration of a first camera having a first sensor, wherein the first initial extrinsic calibration includes a rotation matrix and an initial translation matrix and is based on a first position of the first sensor when the first camera captures an initial image;
determining, using the at least one processor, a first translation of the first sensor based on a second position thereof when the first camera captures a first image, the second position being different than the first position;
determining, using the at least one processor, a first extrinsic translation shift based at least upon the first translation, wherein the first extrinsic translation shift represents a first net translation between the first position and the second position of the first sensor; and
updating, using the at least one processor, the first initial extrinsic calibration to yield a first modified extrinsic calibration for the first camera based on the first extrinsic translation shift, wherein the first modified extrinsic calibration includes the rotation matrix and a first adjusted translation matrix given by a summation of the first extrinsic translation shift with the initial translation matrix.

### Claim 12 (depends on 10)

The vehicle of claim 10, wherein the execution further carrying out operations comprising:
determining, using the at least one processor, a second translation of the first sensor based on a third position thereof when the first camera captures a second image, the third position being different than the first position;
determining, using the at least one processor, a second extrinsic translation shift based at least upon the second translation, wherein the second extrinsic translation shift represents a net translation between the first position and the third position of the first sensor; and
updating, using the at least one processor, the first initial extrinsic calibration to yield a second modified extrinsic calibration for the first camera based on the second extrinsic translation shift, wherein the second modified extrinsic calibration includes the rotation matrix and a second adjusted translation matrix given by a summation of the second extrinsic translation shift with the initial translation matrix.

### Claim 13 (depends on 10)

The vehicle of claim 10, wherein the execution further carrying out operations comprising:
receiving, using the at least one processor, a second initial extrinsic calibration of a second camera having a second sensor, wherein the second initial extrinsic calibration includes a rotation matrix and an initial translation matrix and is based on a first position of the second sensor when the second camera captures an initial image;
determining, using the at least one processor, a second translation of the second sensor based on the second position thereof when the second camera captures a first image, the second position being different than the first position;
determining, using the at least one processor, a second extrinsic translation shift based at least upon the second translation, wherein the second extrinsic translation shift represents a first net translation between the first position and the second position of the second sensor; and
updating, using the at least one processor, the second initial extrinsic calibration to yield a second modified extrinsic calibration for the second camera based on the second extrinsic translation shift, wherein the second modified extrinsic calibration includes the rotation matrix and a second adjusted translation matrix given by the summation of the second extrinsic translation shift with the initial translation matrix.

### Claim 14 (depends on 13)

The vehicle of claim 13, wherein the execution further carrying out operations comprising fusing the data of the first modified extrinsic calibration with data from a third sensor to rectify the first image.

### Claim 15 (depends on 14)

The vehicle of claim 14, wherein the third sensor comprises a LIDAR sensor or a radar sensor.

### Claim 16 (depends on 10)

The vehicle of claim 10, wherein the first camera comprises an IBIS camera.

### Claim 17 (depends on 13)

The vehicle of claim 13, wherein the second camera comprises an IBIS camera.

### Claim 18 (depends on 13)

The vehicle of claim 13, wherein the second sensor is stationary relative to the first sensor.

### Claim 19 (depends on 10)

The vehicle of claim 10, wherein the execution further carrying out operations comprising causing the vehicle to operate based at least in part on output of the first camera.

### Claim 20 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the at least one program including the instructions which, when executed using the at least one processor, cause the first device to further perform operations comprising:
determining, using the at least one processor, a second translation of the first sensor based on a third position thereof when the first camera captures a second image, the third position being different than the first position;
determining, using the at least one processor, a second extrinsic translation shift based at least upon the second translation, wherein the second extrinsic translation shift represents a net translation between the first position and the third position of the first sensor; and
updating, using the at least one processor, the first initial extrinsic calibration to yield a second modified extrinsic calibration for the first camera based on the second extrinsic translation shift, wherein the second modified extrinsic calibration includes the rotation matrix and a second adjusted translation matrix given by a summation of the second extrinsic translation shift with the initial translation matrix.

### Claim 21 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the at least one program including the instructions which, when executed using the at least one processor, cause the first device to further perform operations comprising:
receiving, using the at least one processor, a second initial extrinsic calibration of a second camera having a second sensor, wherein the second initial extrinsic calibration includes a rotation matrix and an initial translation matrix and is based on a first position of the second sensor when the second camera captures an initial image;
determining, using the at least one processor, a second translation of the second sensor based on the second position thereof when the second camera captures a first image, the second position being different than the first position;
determining, using the at least one processor, a second extrinsic translation shift based at least upon the second translation, wherein the second extrinsic translation shift represents a first net translation between the first position and the second position of the second sensor; and
updating, using the at least one processor, the second initial extrinsic calibration to yield a second modified extrinsic calibration for the second camera based on the second extrinsic translation shift, wherein the second modified extrinsic calibration includes the rotation matrix and a second adjusted translation matrix given by the summation of the second extrinsic translation shift with the initial translation matrix.

### Claim 22 (depends on 21)

The non-transitory computer-readable storage medium of claim 21, wherein the at least one program including the instructions which, when executed using the at least one processor, cause the first device to further perform operations comprising fusing the data of the first modified extrinsic calibration with data from a third sensor to rectify the first image.

### Claim 23 (depends on 22)

The non-transitory computer-readable storage medium of claim 22, wherein the third sensor comprises a LIDAR sensor or a radar sensor.

### Claim 24 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the first camera comprises an IBIS camera.

### Claim 25 (depends on 21)

The non-transitory computer-readable storage medium of claim 21, wherein the second camera comprises an IBIS camera.

### Claim 26 (depends on 21)

The non-transitory computer-readable storage medium of claim 21, wherein the second sensor is stationary relative to the first sensor.

### Claim 27 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the at least one program including the instructions which, when executed using the at least one processor, cause the first device to further perform operations comprising causing a vehicle to operate based at least in part on output of the first camera.
