# US 12,002,278 B2 · Object recognition by far infrared camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,278 B2
- Document ID: US12002278B2
- Status: Granted
- Kind: B2
- Application number: 17/624,168
- Filed: 2020-04-16
- Published: 2022-11-17
- Granted: 2024-06-04
- Earliest claimed: 2019-07-03
- Assignees: Hitachi Astemo, Ltd.
- Applicants: Hitachi Astemo, Ltd.
- Inventors: Takeo Shibata
- Web page: https://patentgrep.com/patent/US12002278B2

## CPC

- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G01J 5/10: using electric radiation detectors
- G05D 1/0242: {using non-visible light signals, e.g. IR or UV signals}
- G05D 1/247: using signals provided by artificial sources external to the vehicle, e.g. navigation beacons
- G06F 18/24: Classification techniques
- G06F 18/25: Fusion techniques
- G06T 7/593: from stereo images
- G06V 10/143: Sensing or illuminating at different wavelengths
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- H04N 23/20: for generating image signals from infrared radiation only
- G01J 2005/0077: {Imaging}
- G06F 18/251: {of input or preprocessed data}
- G06T 2207/10012: Stereo images
- G06T 2207/10024: Color image
- G06T 2207/10048: Infrared image
- G06T 2207/20212: Image combination
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Example implementations described herein are directed to integration of far infrared cameras in a vehicle system to detect objects based on relative temperature of objects. Such implementations can improve accuracy when paired, for example, with classification systems that classify objects based on the shape of the object, as both the shape and relative temperature can be used to ensure that the classification is accurate. Further, example implementations can synchronize far infrared cameras with other sensor systems to determine distance, energy, and absolute temperature of an object, which can also be used to enhance classification. Such classifications can then be provided to an advanced driver assistance systems (ADAS), which can control the vehicle system in accordance with the object classification.

## Claims

### Claim 1

A vehicle system, comprising:
one or more far infrared (FIR) cameras;
and a processor, configured to:
classify a first object detected from one or more images of the one or more FIR cameras;
for a second object detected from the one or more images of the one or more FIR cameras having a relative temperature range coinciding with the first object and within an energy range of the first object, recognize the second object as having a classification of the first object,
wherein the energy range of the first object is based on a temperature and a distance to the first object; and
determine an energy range of the second object based on the classification of the first object.

### Claim 2 (depends on 1)

The vehicle system of claim 1, wherein the processor is configured to classify the first object detected from the one or more images of the one or more FIR cameras by:
classify the first object detected from the one or more images of the one or more FIR cameras based on a shape of the first object;
determine a relative temperature range of the first object; and
associate the relative temperature range of the first object to the classification of the first object.

### Claim 3 (depends on 1)

The vehicle system of claim 1, further comprising a sensor system configured to provide image information and distance information, wherein the processor is configured to synchronize the one or more images of the FIR cameras with the image information and distance information from the sensor system, and classify the first object based on the image information and distance information.

### Claim 4 (depends on 1)

The vehicle system of claim 1, further comprising a thermometer configured to measure a temperature of a point in proximity to the vehicle system, wherein the processor is configured to:
determine an absolute temperature range of the first object based on the temperature of the point in proximity of the vehicle system and a relative temperature range of the first object;
for the second object detected from the one or more images of the one or more FIR cameras:
process overlapping ones of the one or more images from multiple ones of the one or more FIR cameras;
convert the relative temperature range of the second object into an absolute temperature range of the second object based on the temperature of the point in proximity to the vehicle system and the relative temperature range of the second object; and
for the second object having the absolute temperature range of the first object, recognize the second object as having the classification of the first object.

### Claim 5 (depends on 1)

The vehicle system of claim 1, wherein the processor is further configured to:
determine energy for the first object based on the temperature and the distance to the first object; and
classify the first object detected from the one or more images of the one or more FIR cameras based on the energy.

### Claim 6 (depends on 1)

The vehicle system of claim 1, wherein the processor is further configured to:
determine a distance range to the second object based on the energy range of the second object and the relative temperature range.

### Claim 7 (depends on 1)

The vehicle system of claim 1, further comprising an advanced driver assistance system (ADAS) configured to control the vehicle system according to the classification of the first object and the second object.

### Claim 8 (depends on 1)

The vehicle system of claim 1, the processor further configured to:
for the second object detected from the one or more images of the one or more FIR cameras not having the relative temperature range coinciding with the first object, recognize the second object based on the relative temperature range in comparison to relative temperature ranges of previous detected objects.

### Claim 9

A non-transitory computer readable medium, storing instructions for executing a process for a vehicle system comprising one or more far infrared (FIR) cameras, the instructions comprising:
classifying a first object detected from one or more images of the one or more FIR cameras;
for a second object detected from the one or more images of the one or more FIR cameras having a relative temperature range coinciding with the first object and within an energy range of the first object, recognizing the second object as having a classification of the first object,
wherein the energy range of the first object is based on a temperature and a distance to the first object; and
determining an energy range of the second object based on the classification of the first object.

### Claim 10 (depends on 9)

The non-transitory computer readable medium of claim 9, wherein the classifying the first object detected from the one or more images of the one or more FIR cameras comprises:
classifying the first object detected from the one or more images of the one or more FIR cameras based on a shape of the first object;
determining a relative temperature range of the first object; and
associating the relative temperature range of the first object to the classification of the first object.

### Claim 11 (depends on 9)

The non-transitory computer readable medium of claim 9, the vehicle system further comprising a sensor system configured to provide image information and distance information, wherein the instructions further comprise synchronizing the one or more images of the FIR cameras with the image information and distance information from the sensor system, and classifying the first object based on the image information and distance information.

### Claim 12 (depends on 9)

The non-transitory computer readable medium of claim 9, the vehicle system further comprising a thermometer configured to measure a temperature of a point in proximity to the vehicle system, the instructions further comprising:
determining an absolute temperature range of the first object based on the temperature of the point in proximity of the vehicle system and a relative temperature range of the first object;
for the second object detected from the one or more images of the one or more FIR cameras:
processing overlapping ones of the one or more images from multiple ones of the one or more FIR cameras;
converting the relative temperature range of the second object into an absolute temperature range of the second object based on the temperature of the point in proximity to the vehicle system and the relative temperature range of the second object; and
for the second object having the absolute temperature range of the first object, recognizing the second object as having the classification of the first object.

### Claim 13 (depends on 9)

The non-transitory computer readable medium of claim 9, the instructions further comprising:
determining energy for the first object based on the temperature and the distance to the first object; and
classifying the first object detected from the one or more images of the one or more FIR cameras based on the energy.

### Claim 14 (depends on 9)

The non-transitory computer readable medium of claim 9, the instructions further comprising:
determining a distance range to the second object based on the energy range of the second object and the relative temperature range.

### Claim 15 (depends on 9)

The non-transitory computer readable medium of claim 9, the instructions further comprising:
for the second object detected from the one or more images of the one or more FIR cameras not having the relative temperature range coinciding with the first object, recognize the second object based on the relative temperature range in comparison to relative temperature ranges of previous detected objects.