# US 11,122,248 B1 · Stereo vision with weakly aligned heterogeneous cameras

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,122,248 B1
- Document ID: US11122248B1
- Status: Granted
- Kind: B1
- Application number: 16/932,830
- Filed: 2020-07-20
- Granted: 2021-09-14
- Earliest claimed: 2020-07-20
- Assignees: Black Sesame International Holding Limited
- Applicants: Black Sesame International Holding Limited
- Inventors: Zuoguan Wang; Jizhang Shan
- Web page: https://patentgrep.com/patent/US11122248B1

## CPC

- H04N 13/128: Adjusting depth or disparity
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/246: Calibration of cameras
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method of depth estimation utilizing heterogeneous cameras, comprising, homogenizing a first camera image and a second camera image based on a first camera calibration dataset and a second camera calibration dataset respectively, wherein the first camera image and second camera image are distortion corrected and are zoom compensated, determining an initial image pair rectification transformation matrix of the homogenized first camera image and the homogenized second camera image, determining a delta image pair rectification transformation matrix based on the initial image pair rectification transformation matrix, determining a final image pair rectification transformation matrix based on the initial image pair rectification transformation matrix and the delta image pair rectification transformation matrix resulting in a final rectified image pair and disparity mapping the final rectified image pair based on a depth net regression.

## Claims

### Claim 1

A method of depth estimation utilizing two heterogeneous cameras, comprising:
calibrating a first camera having a first camera calibration dataset;
calibrating a second camera having a second camera calibration dataset, the second camera having at least one of a different focal length, a different field of view and a different pixel count than the first camera;
distortion correcting between a first camera image and a second camera image based on the first camera calibration dataset and the second camera calibration dataset;
focal length compensating between the first camera image and the second camera image based on the first camera calibration dataset and the second camera calibration dataset;
disparity mapping the distortion corrected and focal length compensated first camera image based on a depth net regression;
determining an initial pose based on the first camera calibration dataset and the second camera calibration dataset;
pose mapping using a pose net the first camera image and the second camera image and the initial pose and outputting a delta pose;
warping the delta pose, the disparity mapping and the first camera image; and
reconstructing the warping and the distortion corrected and focal length compensated second camera image to minimize a reconstruction error.

### Claim 2 (depends on 1)

The method of depth estimation of claim 1, wherein the pose mapping is performed online.

### Claim 3 (depends on 1)

The method of depth estimation of claim 1, wherein the pose mapping is tracked.

### Claim 4 (depends on 1)

The method of depth estimation of claim 1, wherein the first camera and the second camera are weakly aligned.

### Claim 5 (depends on 1)

The method of depth estimation of claim 1, wherein distortion correction is performed before image pair rectification.

### Claim 6 (depends on 1)

The method of depth estimation of claim 1, wherein focal length compensation is performed before image pair rectification.

### Claim 7 (depends on 1)

The method of depth estimation of claim 1, wherein a neural network is used to regress a drift of a pose map transformation matrix.

### Claim 8 (depends on 1)

The method of depth estimation of claim 1, wherein a neural network is utilized to disparity map the first camera image.
