# US 2025/0095380 A1 · Self extrinsic self-calibration via geometrically consistent self-supervised depth and ego-motion learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095380 A1
- Document ID: US20250095380A1
- Status: Published application
- Kind: A1
- Application number: 18/469,373
- Filed: 2023-09-18
- Published: 2025-03-20
- Earliest claimed: 2023-09-18
- Assignees: Toyota Research Institute, Inc.; Toyota Jidosha Kabushiki Kaisha
- Applicants: Toyota Research Institute, Inc.
- Inventors: TAKAYUKI KANAI; Vitor Campagnolo Guizilini; Rares A. Ambrus; Adrien Gaidon; Igor Vasiljevic
- Web page: https://patentgrep.com/patent/US20250095380A1

## CPC

- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- B60W 40/105: Speed
- B60W 2420/403: Image sensing, e.g. optical camera
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT

## Abstract

Systems and methods described herein relate to self-supervised scale-aware learning of camera extrinsic parameters. One embodiment processes instantaneous velocity between a target image and a context image captured by a first camera; jointly training a depth network and pose network based on scaling by the instantaneous velocity; produce depth map using the depth network; produce ego-motion of the first camera using the pose network; generate synthesized image from the target image using a reprojection operation based on the depth map, the ego-motion, the context image and camera intrinsics; determine photometric loss by comparing the synthesized image to the target image; generate photometric consistency constraint using a gradient from the photometric loss; determine pose consistency constraint between the first camera and a second camera; and optimize the photometric consistency constraint, the pose consistency constraint, the depth network and the pose network to generate estimated extrinsic parameters.

## Claims

### Claim 1

A system for self-supervised scale-aware learning of camera extrinsic parameters, the system comprising:
one or more processors; and
a memory communicably coupled to the one or more processors and including instructions that when executed by the one or more processors cause the one or more processors to:
process instantaneous velocity between a target image and a context image captured by a first camera;
jointly train a depth network and pose network based on scaling by the instantaneous velocity, wherein the depth network produces predicted depth maps for the target image and the pose network predicts the relative rigid-body transformation between the target image and the context image;
process, using the depth network, the target image to produce a depth map;
process, using the pose network and the instantaneous velocity, the target image and the context image to produce an ego-motion of the first camera;
generate a synthesized image from the target image using a reprojection operation based on the depth map, the ego-motion and the context image;
determine photometric loss by comparing the synthesized image to the target image;
generate photometric consistency constraint using a gradient from the photometric loss;
determine pose consistency constraint between the first camera and a second camera; and
optimize the photometric consistency constraint, the pose consistency constraint, the depth network and the pose network to generate estimated extrinsic parameters.

### Claim 2 (depends on 1)

The system of claim 1, further comprising instructions that cause the one or more processors to:
regularize the depth map to provide smoothing of depth estimations.

### Claim 3 (depends on 1)

The system of claim 1, wherein the synthesized image is generated from the target image using the reprojection operation further based on camera intrinsics and the camera intrinsics are associated with a parametric camera model.

### Claim 4 (depends on 1)

The system of claim 1, wherein the scaling by the instantaneous velocity is used to train the depth network to produce metrically accurate and scale-aware depth maps.

### Claim 5 (depends on 1)

The system of claim 1, wherein the scaling by the instantaneous velocity is used to train the pose network to produce metrically accurate and scale-aware ego-motion.

### Claim 6 (depends on 1)

The system of claim 1, wherein the determine pose consistency constraint between the first camera and a second camera comprises:
processing, using the pose network and the instantaneous velocity, a second target image and a second context image to produce a second ego-motion, wherein the second target image and the second context image are captured by the second camera;
converting the second ego-motion of the second camera to a coordinate frame of the first camera;
in response to converting the second ego-motion to the coordinate frame of the first camera, constraining translation vectors of the first and second cameras to determine a translation consistency loss;
in response to converting the second ego-motion to the coordinate frame of the first camera, converting rotation matrices of the first and second cameras to determine a rotation consistency loss; and
determining a pose consistency loss by combining the translation consistency loss and the rotation consistency loss.

### Claim 7 (depends on 6)

The system of claim 6, further comprising instructions that cause the one or more processors to:
train the depth network and the pose network according to the pose consistency loss to generate metrically accurate and scale-aware depth maps and ego-motion.

### Claim 8 (depends on 1)

The system of claim 1, further comprising instructions that cause the one or more processors to:
receive a first set of images captured by the first camera and a second set of images captured by the second camera;
warp images between the first set and the second set;
generate spatio-temporal contexts by processing the warped images across a spatial axis and a temporal axis; and
jointly train the depth network and the pose network with the spatio-temporal contexts.

### Claim 9 (depends on 1)

The system of claim 1, wherein the first camera and the second camera are rigged to a vehicle.

### Claim 10 (depends on 9)

The system of claim 9, wherein the ego-motion of the first camera is identical to an ego-motion of the second camera.

### Claim 11 (depends on 3)

The system of claim 3, further comprising instructions that cause the one or more processors to:
update the camera intrinsics on a per-image sequence basis using the gradient from the photometric loss.

### Claim 12

A non-transitory computer-readable medium for self-supervised scale-aware learning of camera extrinsic parameters and storing instructions that when executed by one or more processors cause the one or more processors to:
process instantaneous velocity between a target image and a context image captured by a first camera;
jointly train a depth network and pose network based on scaling by the instantaneous velocity, wherein the depth network produces predicted depth maps for the target image and the pose network predicts the relative rigid-body transformation between the target image and the context image;
process, using the depth network, the target image to produce a depth map;
process, using the pose network and the instantaneous velocity, the target image and the context image to produce an ego-motion of the first camera;
generate a synthesized image from the target image using a reprojection operation based on the depth map, the ego-motion and the context image;
determine photometric loss by comparing the synthesized image to the target image;
generate photometric consistency constraint using a gradient from the photometric loss;
determine pose consistency constraint between the first camera and a second camera; and
optimize the photometric consistency constraint, the pose consistency constraint, the depth network and the pose network to generate estimated extrinsic parameters.

### Claim 13 (depends on 12)

The non-transitory computer-readable medium of claim 12, further comprising instructions that cause the one or more processors to:
regularize the depth map to provide smoothing of depth estimations.

### Claim 14 (depends on 12)

The non-transitory computer-readable medium of claim 12, wherein the scaling by the instantaneous velocity is used to train the depth network to produce metrically accurate and scale-aware depth maps.

### Claim 15 (depends on 12)

The non-transitory computer-readable medium of claim 12, wherein the scaling by the instantaneous velocity is used to train the pose network to produce metrically accurate and scale-aware ego-motion.

### Claim 16 (depends on 12)

The non-transitory computer-readable medium of claim 12, wherein the determine pose consistency constraint between the first camera and a second camera comprises:
processing, using the pose network and the instantaneous velocity, a second target image and a second context image to produce a second ego-motion, wherein the second target image and the second context image are captured by the second camera;
converting the second ego-motion of the second camera to a coordinate frame of the first camera;
in response to converting the second ego-motion to the coordinate frame of the first camera, constraining translation vectors of the first and second cameras to determine a translation consistency loss;
in response to converting the second ego-motion to the coordinate frame of the first camera, converting rotation matrices of the first and second cameras to determine a rotation consistency loss; and
determining a pose consistency loss by combining the translation consistency loss and the rotation consistency loss.

### Claim 17 (depends on 12)

The non-transitory computer-readable medium of claim 12, further comprising instructions that cause the one or more processors to:
train the depth network and the pose network according to the pose consistency loss to generate metrically accurate and scale-aware depth maps and ego-motion.

### Claim 18 (depends on 12)

The non-transitory computer-readable medium of claim 12, further comprising instructions that cause the one or more processors to:
receive a first set of images captured by the first camera and a second set of images captured by the second camera;
warp images between the first set and the second set;
generate spatio-temporal contexts by processing the warped images across a spatial axis and a temporal axis; and
jointly train the depth network and the pose network with the spatio-temporal contexts.

### Claim 19 (depends on 12)

The non-transitory computer-readable medium of claim 12, wherein the synthesized image is generated from the target image using the reprojection operation further based on camera intrinsics and the camera intrinsics are associated with a parametric camera model, and the medium further comprises instructions that cause the one or more processors to:
update the camera intrinsics on a per-image sequence basis using the gradient from the photometric loss.

### Claim 20

A method of self-supervised scale-aware learning of camera extrinsic parameters, the method comprising:
processing instantaneous velocity information between a target image and a context image captured by a first camera;
jointly training a depth network and pose network based on scaling by the instantaneous velocity, wherein the depth network produces predicted depth maps for the target image and the pose network predicts the relative rigid-body transformation between the target image and the context image;
processing, using the depth network, the target image to produce a depth map;
processing, using the pose network and the instantaneous velocity, the target image and the context image to produce an ego-motion of the first camera;
generating a synthesized image from the target image using a reprojection operation based on the depth map, the ego-motion and the context image;
determining photometric loss by comparing the synthesized image to the target image;
generating photometric consistency constraint using a gradient from the photometric loss;
determining pose consistency constraint between the first camera and a second camera; and
optimizing the photometric consistency constraint, the pose consistency constraint, the depth network and the pose network to generate estimated extrinsic parameters.
