# US 2022/0277485 A1 · Cross spectral feature mapping for camera calibration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277485 A1
- Document ID: US20220277485A1
- Status: Published application
- Kind: A1
- Application number: 17/597,428
- Filed: 2021-01-19
- Published: 2022-09-01
- Earliest claimed: 2021-01-19
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Supreeth Achar; Daniel Goldman
- Web page: https://patentgrep.com/patent/US20220277485A1

## CPC

- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 7/85: {Stereo camera calibration}
- G06T 7/74: {involving reference images or patches}
- G06V 10/48: by mapping characteristic values of the pattern into a parameter space, e.g. Hough transformation
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- H04N 13/246: Calibration of cameras
- H04N 23/56: provided with illuminating means
- G06T 2207/10012: Stereo images
- G06T 2207/10024: Color image
- G06T 2207/10048: Infrared image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method including capturing a first image of a real-world scene by a first camera sensitive to a first spectrum of light, the first camera having a first light source, capturing a second image of the real-world scene by a second camera sensitive to a second spectrum of light, the second camera having a second light source, identifying at least one feature in the first image, identifying, using a machine learning (ML) model, at least one feature in the second image that matches the at least one feature identified in the first image, mapping pixels in the first image and the second image to rays in a three-dimensional (3D) space based on the matched at least one feature, and calibrating the first camera and the second camera based on the mapping.

## Claims

### Claim 1

A method comprising:
capturing a first image of a real-world scene by a first camera sensitive to a first spectrum of light, the first camera having a first light source;
capturing a second image of the real-world scene by a second camera sensitive to a second spectrum of light, the second camera having a second light source;
identifying at least one feature in the first image;
identifying, using a machine learning (ML) model, at least one feature in the second image that matches the at least one feature identified in the first image;
mapping pixels in the first image and the second image to rays in a three-dimensional (3D) space based on the matched at least one feature; and
calibrating the first camera and the second camera based on the mapping.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first camera is a near infrared (NIR) camera, and the second camera is a visible light camera.

### Claim 3 (depends on 1)

The method of claim 1, wherein a ML model is used to identify the at least one feature in the first image.

### Claim 4 (depends on 1)

The method of claim 1, wherein an algorithm is used to identify the at least one feature in the first image.

### Claim 5 (depends on 1)

The method of claim 1, wherein
a ML model is used to match the at least one feature in the first image to the at least one feature in the second image, and
at least one pixel of the second image is assigned a score based on a likelihood that the at least one pixel of the second image matches a pixel of the at least one feature in the first image.

### Claim 6 (depends on 1)

The method of claim 1, wherein
an algorithm is used to match the at least one feature in the first image to the at least one feature in the second image,
at least one pixel of the second image is assigned a score based on a likelihood that the at least one pixel of the second image matches a pixel of the at least one feature in the first image, and
the at least one pixel of the second image is assigned a direction based on a prediction of a position of a target pixel.

### Claim 7 (depends on 1)

The method of claim 1, wherein
an algorithm is used to identify the at least one feature in the first image, and
the matching of the at least one feature in the first image to the at least one feature in the second image includes
selecting, using a first ML model, a candidate feature from the at least one feature in the first image,
matching at least one pixel in the second image to a pixel of the candidate feature,
assigning a score to the matched at least one pixel of the second image based on a likelihood that the at least one pixel matches one of the at least one feature in the first image,
predicting a direction of a position of a target pixel using a second ML model, and
assigning the direction to the matched at least one pixel of the second image.

### Claim 8 (depends on 7)

The method of claim 7, wherein the calibrating of the first camera and the second camera is based on the matched at least one pixel of the second image associated with the candidate feature with the highest score and the direction of the matched at least one pixel of the second image with the highest score, the direction being based on the matched at least one pixel of the second image with the highest score and neighboring pixels.

### Claim 9 (depends on 1)

The method of claim 1, further comprising selecting at least one search window in the second image based on a previous calibration.

### Claim 10 (depends on 1)

The method of claim 1, wherein the machine learning model(s) are trained on data captured from a calibrated multi camera system.

### Claim 11

A three-dimensional (3D) teleconferencing system comprising:
a memory including code segments representing a plurality of computer instructions; and
a processor configured to execute the code segments, the computer instructions including:
capturing a first image of a real-world scene by a first camera sensitive to a first spectrum of light, the first camera having a first light source;
capturing a second image of the real-world scene by a second camera sensitive to a second spectrum of light, the second camera having a second light source;
identifying at least one feature in the first image;
identifying, using a machine learning (ML) model, at least one feature in the second image that matches the at least one feature identified in the first image;
mapping pixels in the first image and the second image to rays in a three-dimensional (3D) space based on the matched at least one feature; and
calibrating the first camera and the second camera based on the mapping.

### Claim 12 (depends on 11)

The system of claim 11, wherein the first camera is a near infrared (NIR) camera, and the second camera is a visible light camera.

### Claim 13 (depends on 11)

The system of claim 11, wherein a ML model is used to identify the at least one feature in the first image.

### Claim 14 (depends on 11)

The system of claim 11, wherein an algorithm is used to identify the at least one feature in the first image.

### Claim 15 (depends on 11)

The system of claim 11, wherein
a ML model is used to match the at least one feature in the first image to the at least one feature in the second image, and
at least one pixel of the second image is assigned a score based on a likelihood that at least one pixel of the second image matches a pixel of the at least one feature in the first image.

### Claim 16 (depends on 11)

The system of claim 11, wherein
an algorithm is used to match the at least one feature in the first image to the at least one feature in the second image,
at least one pixel of the second image is assigned a score based on a likelihood that the at least one pixel of the second image matches a pixel of the at least one feature in the first image, and
the at least one pixel of the second image is assigned a direction based on a prediction of a position of a target pixel.

### Claim 17 (depends on 11)

The system of claim 11, wherein
an algorithm is used to identify the at least one feature in the first image, and
the matching of the at least one feature in the first image to the at least one feature in the second image includes
selecting, using a first ML model, a candidate feature from the at least one feature in the first image,
matching at least one pixel of the second image to the candidate feature,
assigning a score to the matched at least one pixel of the second image based on a likelihood that the at least one pixel of the second image matches one of the at least one feature in the first image,
predicting a direction of a position of a target pixel using a second ML model, and
assigning the direction to the matched at least one pixel of the second image.

### Claim 18 (depends on 17)

The system of claim 17, wherein the calibrating of the first camera and the second camera is based on the matched at least one pixel of the second image with the highest score and the direction of the matched at least one pixel of the second image with the highest score, the direction being based on the matched at least one pixel of the second image with the highest score and a direction of neighboring pixels.

### Claim 19 (depends on 11)

The system of claim 11, further comprising selecting at least one search window in the second image based on a previous calibration.

### Claim 20 (depends on 11)

The system of claim 11, wherein the machine learning model(s) are trained on data captured from a calibrated multi camera system.

### Claim 21

A non-transitory computer readable medium containing instructions that when executed cause a processor of a computer system to perform steps comprising:
capturing a first image of a real-world scene by a first camera sensitive to a first spectrum of light, the first camera having a first light source;
capturing a second image of the real-world scene by a second camera sensitive to a second spectrum of light, the second camera having a second light source;
identifying at least one feature in the first image;
identifying, using a machine learning (ML) model, at least one feature in the second image that matches the at least one feature identified in the first image;
mapping pixels in the first image and the second image to rays in a three-dimensional (3D) space based on the matched at least one feature; and
calibrating the first camera and the second camera based on the mapping.

### Claim 22 (depends on 21)

The non-transitory computer readable medium of claim 21, wherein
an algorithm is used to identify the at least one feature in the first image, and
the matching of the at least one feature in the first image to the at least one feature in the second image includes
selecting, using a first ML model, a candidate feature from the at least one feature in the first image,
matching at least one pixel of the second image to a pixel of the candidate feature,
assigning a score to the matched at least one pixel of the second image based on a likelihood that the at least one pixel of the second image matches one of the at least one feature in the first image,
predicting a direction of a position of a target pixel using a second ML model, and
assigning the direction to the matched at least one pixel of the second image.

### Claim 23 (depends on 21)

The non-transitory computer readable medium of claim 21, wherein the first spectrum of light and the second spectrum of light are a same spectrum of light.
