# US 2022/0021861 A1 · Stereo vision with weakly aligned heterogeneous cameras

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0021861 A1
- Document ID: US20220021861A1
- Status: Published application
- Kind: A1
- Application number: 17/342,017
- Filed: 2021-06-08
- Published: 2022-01-20
- Earliest claimed: 2020-07-20
- Assignees: Black Sesame International Holding Limited
- Applicants: Black Sesame International Holding Limited
- Inventors: Zuoguan Wang; Jizhang Shan
- Web page: https://patentgrep.com/patent/US20220021861A1

## CPC

- H04N 13/128: Adjusting depth or disparity
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/246: Calibration of cameras
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method of depth estimation utilizing heterogeneous cameras, comprising, homogenizing a first camera image and a second camera image based on a first camera calibration dataset and a second camera calibration dataset respectively, wherein the first camera image and second camera image are distortion corrected and are zoom compensated, determining an initial image pair rectification transformation matrix of the homogenized first camera image and the homogenized second camera image, determining a delta image pair rectification transformation matrix based on the initial image pair rectification transformation matrix, determining, a final image pair rectification transformation matrix based on the initial image pair rectification transformation matrix and the delta image pair rectification transformation matrix resulting in a final rectified image pair and disparity mapping the final rectified image pair based on a depth net regression.

## Claims

### Claim 1

A method of depth estimation utilizing two heterogeneous cameras, comprising:
homogenizing a first camera image and a second camera image based on a first camera calibration dataset and a second camera calibration dataset respectively, wherein the first camera image and second camera image are distortion corrected and are zoom compensated;
determining an initial image pair rectification transformation matrix of the homogenized first camera image and the homogenized second camera image;
determining a delta image pair rectification transformation matrix based on the initial image pair rectification transformation matrix;
determining a final image pair rectification transformation matrix based on the initial image pair rectification transformation matrix and the delta image pair rectification transformation matrix resulting in a final rectified image pair; and
disparity mapping the final rectified image pair based on a depth net regression.

### Claim 2 (depends on 1)

The method of depth estimation of claim 1, wherein a first camera and a second camera are weakly aligned.

### Claim 3 (depends on 1)

The method of depth estimation of claim 1, wherein distortion correction performed before image pair rectification.

### Claim 4 (depends on 1)

The method of depth estimation of claim 1, wherein focal length compensation is performed before image pair rectification.

### Claim 5 (depends on 1)

The method, of depth estimation of claim 1, wherein a neural network is used to regress a drift of the delta image pair rectification transformation matrix.

### Claim 6 (depends on 1)

The method of depth estimation of claim 1, wherein a neural network is utilized to disparity map the final rectified image pair.

### Claim 7

A method of depth estimation utilizing two heterogeneous cameras, comprising:
calibrating a first camera having a first camera calibration dataset;
calibrating a second camera having a second camera calibration dataset, the second camera having at least one of a different focal length, a different field of view and a different pixel count than the first camera;
distortion correcting between a first camera image and a second camera image based on the first camera calibration dataset and the second camera calibration dataset;
focal length compensating between the first camera image and the second camera image based on the first camera calibration dataset and the second camera calibration dataset;
rectifying an image pair between the distortion corrected and focal length compensated first camera image and the second camera image based on a transformation matrix regression; and
disparity mapping the rectified image pair based on a depth net regression.

### Claim 8 (depends on 7)

The method of depth estimation of claim 7, wherein the first camera and the second camera are weakly aligned.

### Claim 9 (depends on 7)

The method of depth estimation of claim 7, wherein distortion correction is performed before image pair rectification.

### Claim 10 (depends on 7)

The method of depth estimation of claim 7, wherein focal length compensation is performed before image pair rectification.

### Claim 11 (depends on 7)

The method of depth estimation of claim 7, wherein a neural network is used to regress a drift of a rectifying transformation matrix of the rectified image pair.

### Claim 12 (depends on 7)

The method of depth estimation of claim 7, Wherein a neural network is utilized to disparity map the rectified image pair.
