# US 2021/0342648 A1 · Annotation of infrared images for machine learning using beamsplitter-based camera system and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342648 A1
- Document ID: US20210342648A1
- Status: Published application
- Kind: A1
- Application number: 17/246,303
- Filed: 2021-04-30
- Published: 2021-11-04
- Earliest claimed: 2020-04-30
- Assignees: FLIR Commercial Systems, Inc.
- Applicants: FLIR Commercial Systems, Inc.
- Inventors: Austin A. Richards; Andres Prieto-Moreno
- Web page: https://patentgrep.com/patent/US20210342648A1

## CPC

- G06V 10/143: Sensing or illuminating at different wavelengths
- G06F 18/24: Classification techniques
- G06T 7/37: using transform domain methods
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- H04N 23/11: for generating image signals from visible and infrared light wavelengths
- H04N 23/611: where the recognised objects include parts of the human body
- H04N 23/62: Control of parameters via user interfaces
- G06T 2207/10048: Infrared image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Systems and methods include an infrared camera configured to capture an infrared image of a scene, a visible light camera configured to capture a visible light image of the scene, and a logic device configured to simultaneously capture a pair of images of the scene comprising the infrared image of the scene and the visible image of the scene, align the pair of images so that a pixel location in one of the pair of images has a corresponding pixel location in the other image, classify the visible image, annotate the infrared image based, at least in part, on the classification of the visible image, and add the annotated infrared image to a neural network training dataset for use in training a neural network for infrared image classification.

## Claims

### Claim 1

A method comprising:
simultaneously capturing a pair of images of a scene comprising an infrared image of the scene and a visible image of the scene;
aligning the pair of images so that a pixel location in one of the pair of images has a corresponding pixel location in the other image;
classifying the visible image;
annotating the infrared image based, at least in part, on the classification of the visible image; and
adding the annotated infrared image to a neural network training dataset for use in training a neural network for infrared image classification.

### Claim 2 (depends on 1)

The method of claim 1, wherein simultaneously capturing the pair of images of the scene further comprises capturing the infrared image using an infrared camera and capturing the visible image using a visible light camera, wherein the infrared camera and the visible light camera are configured to capture the pair of images of the scene from substantially the same perspective.

### Claim 3 (depends on 1)

The method of claim 1, wherein simultaneously capturing the pair of images of the scene further comprises capturing the pair of images of the scene through a beamsplitter arranged to reflect a first image of the scene towards a first camera and pass through a second image of the scene to a second camera.

### Claim 4 (depends on 1)

The method of claim 1, wherein aligning the pair of images so that the pixel location in one of the pair of images has a corresponding pixel location in the other image further comprises determining a transformation between the pair of images so that the pixel location in each image represents the same location in the scene; and applying the transformation to the image pair to align the images.

### Claim 5 (depends on 1)

The method of claim 1, wherein classifying the visible image comprises receiving an image classification from a user and/or automatically generating a visible image classification using a trained neural network.

### Claim 6 (depends on 1)

The method of claim 1, further comprising classifying the infrared image by receiving an infrared image classification from a user and/or automatically generating an infrared image classification using a trained neural network.

### Claim 7 (depends on 1)

The method of claim 1, wherein the infrared image is annotated based, at least in part, on a classification of the infrared image.

### Claim 8 (depends on 1)

The method of claim 1, wherein annotating the infrared image comprises:
presenting a user interface displaying the visible image, the infrared image and corresponding annotations; and
receiving input from the user modifying one or more annotation.

### Claim 9 (depends on 1)

The method of claim 1, wherein annotating the infrared image comprises applying visible image classification information to the infrared image.

### Claim 10 (depends on 1)

The method of claim 1, further comprising training a neural network to classify an infrared image using the neural network training dataset including the annotated infrared image.

### Claim 11

A system comprising:
an infrared camera configured to capture an infrared image of a scene;
a visible light camera configured to capture a visible light image of the scene; and
a logic device configured to:
simultaneously capture a pair of images of the scene comprising the infrared image of the scene and the visible image of the scene;
align the pair of images so that a pixel location in one of the pair of images has a corresponding pixel location in the other image;
classify the visible image;
annotate the infrared image based, at least in part, on the classification of the visible image; and
add the annotated infrared image to a neural network training dataset for use in training a neural network for infrared image classification.

### Claim 12 (depends on 11)

The system of claim 11, wherein the logic device is configured to simultaneously capture the pair of images of the scene by capturing the infrared image using the infrared camera and capturing the visible image using the visible light camera, wherein the infrared camera and the visible light camera are configured to capture the pair of images of the scene from substantially the same perspective.

### Claim 13 (depends on 11)

The system of claim 11, further comprising a beamsplitter arranged to reflect a first image of the scene towards the infrared camera and pass through a second image of the scene to the visible light camera; and
wherein the logic device is configured to simultaneously capture the pair of images of the scene by capturing the pair of images of the scene through the beamsplitter.

### Claim 14 (depends on 11)

The system of claim 11, wherein the logic device is configured to align the pair of images wherein the pixel location in one of the pair of images has a corresponding pixel location in the other image further by determining a transformation between the pair of images so that the pixel location in each image represents the same location in the scene; and applying the transformation to the image pair to align the images.

### Claim 15 (depends on 11)

The system of claim 11, wherein the logic device is configured to classify the visible image by receiving an image classification from a user and/or automatically generating a visible image classification using a trained neural network.

### Claim 16 (depends on 11)

The system of claim 11, wherein the logic device is further configured to classify the infrared image by receiving an infrared image classification from a user and/or automatically generating an infrared image classification using a trained neural network.

### Claim 17 (depends on 11)

The system of claim 11, wherein the logic device is further configured to annotate the infrared image based, at least in part, on a classification of the infrared image.

### Claim 18 (depends on 11)

The system of claim 11, wherein the logic device is further configured to annotate the infrared image by presenting a user interface displaying the visible image, the infrared image and corresponding annotations; and receiving input from a user modifying one or more annotation.

### Claim 19 (depends on 11)

The system of claim 11, wherein the logic device is further configured to annotate the infrared image by applying visible image classification information to the infrared image.

### Claim 20 (depends on 11)

The system of claim 11, wherein the logic device is further configured to train a neural network to classify an infrared image using the neural network training dataset including the annotated infrared image.
