# US 2020/0396379 A1 · Smart Sensor Measurement System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0396379 A1
- Document ID: US20200396379A1
- Status: Published application
- Kind: A1
- Application number: 16/438,655
- Filed: 2019-06-12
- Published: 2020-12-17
- Earliest claimed: 2019-06-12
- Assignees: Air Products and Chemicals, Inc.
- Applicants: Air Products and Chemicals, Inc.
- Inventors: Avishek Guha; Erdem Arslan
- Web page: https://patentgrep.com/patent/US20200396379A1

## CPC

- G01J 5/0859: {Sighting arrangements, e.g. cameras}
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G01J 5/802: {by correcting for emissivity}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- H04N 23/11: for generating image signals from visible and infrared light wavelengths
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G01J 2005/0077: {Imaging}
- G06T 2207/10048: Infrared image
- G06T 2207/20084: Artificial neural networks [ANN]
- G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A first camera captures a first image of a scene, and a second camera captures a second image capturing thermal distribution across the scene. The first image is processed to detect one or more objects of interest in the scene and to identify respective surface characteristics of the objects. The first image is overlaid with the second image to identify regions of interest in the second image corresponding to the objects of interest in the first image. Values of pixels that belong to respective regions of interest in the second image are converted to temperature values using respective conversion functions that reflect respective emissivity values determined by the identified respective surface characteristics of the corresponding objects of interest in the first image. The temperature values are analyzed to monitor thermal conditions of the one or more objects of interest in the scene.

## Claims

### Claim 1

A method for processing thermal image data in a sensor device, the method comprising:
processing, with a processor of the sensor device, a first image of a scene to detect one or more objects of interest in the scene, wherein detecting the one or more objects of interest includes identifying respective surface characteristics of the one or more objects of interest detected in the scene;
overlaying, with the processor of the sensor device, the first image of the scene with a second image of the scene, the second image capturing thermal distribution across the scene, to identify one or more regions of interest in the second image of the scene corresponding to the one or more objects of interest detected in the first image of the scene;
converting, with the processor of the sensor device, values of pixels in the one or more regions of interest identified in the second image of the scene to temperature values, including converting values of pixels that belong to respective regions of interest, among the one or more regions of interest in the second image of the scene, using respective conversion functions that reflect respective emissivity values of the one or more objects of interest detected in the first image scene, the emissivity values determined by the identified respective surface characteristics of the one or more objects of interest detected in the first image of the scene; and
analyzing, with the processor of the sensor device, the temperature values to monitor thermal conditions of the one or more objects of interest in the scene.

### Claim 2 (depends on 1)

The method of claim 1, wherein processing the first image of the scene comprises using template matching to detect the one or more objects of interest in the first image of the scene, including matching a template image of a particular object of interest in the scene across the first image to identify a matching region in the first image.

### Claim 3 (depends on 1)

The method of claim 1, wherein processing the first image comprises using a neural network trained to detect the one or more objects of interest in the scene.

### Claim 4 (depends on 1)

The method of claim 1, further comprising retrieving, from a memory, respective calibration files specifying the respective conversion functions that reflect respective emissivity values of the one or more objects of interest detected in the scene, the respective calibration files being retrieved based on the identified respective surface characteristics of the one or more objects of interest detected in the scene.

### Claim 5 (depends on 4)

The method of claim 4, wherein
processing the first image includes detecting at least i) a first object of interest in the scene, the first object of interest having first surface characteristics and ii) a second object of interest in the scene, the second object of interest having second surface characteristics different from the first surface characteristics, and
retrieving the respective calibration files includes i) retrieving a first calibration file specifying a first conversion function that is based on a first emissivity value corresponding to the first object of interest and ii) retrieving a second calibration file specifying a second conversion function that reflects a second emissivity value of the second object of interest, the second emissivity value being different from the first emissivity value.

### Claim 6 (depends on 1)

The method of claim 1, further comprising transmitting the temperature values from the sensor device via a communication network to a server device for further processing.

### Claim 7 (depends on 1)

The method of claim 1, wherein
the first image of the scene is a visible image captured by a visible light camera, and
the second image of the scene is an infrared (IR) image captured by an IR camera.

### Claim 8

A sensor device, comprising
a first camera configured to capture a first image of a scene,
a second camera configured to capture a second image of the scene, the second image capturing thermal distribution across the scene, and
a processor coupled to the first camera and the second camera, the processor configured to
process the first image of the scene to detect one or more objects of interest in the scene, wherein detecting the one or more objects of interest includes identifying respective surface characteristics of the one or more objects of interest detected in the scene;
overlay the first image of the scene with the second image of the scene to identify one or more regions of interest in the second image of the scene corresponding to the one or more objects of interest detected in the first image of the scene;
convert values of pixels in the one or more regions of interest identified in the second image of the scene to temperature values, including converting values of pixels that belong to respective regions of interest, among the one or more regions of interest in the second image of the scene, using respective conversion functions that reflect respective emissivity values of the one or more objects of interest detected in the first image scene, the emissivity values determined by the identified respective surface characteristics of the one or more objects of interest detected in the first image of the scene; and
analyze the temperature values to monitor thermal conditions of the one or more objects of interest in the scene.

### Claim 9 (depends on 8)

The sensor device of claim 8, wherein the processor is configured to process the first image of the scene using template matching to detect the one or more objects of interest in the scene, the processor being configured to match a template image of a particular object of interest in the scene across the first image to identify a matching region in the first image.

### Claim 10 (depends on 8)

The sensor device of claim 8, wherein the processor is configured to process the first image comprises using a neural network trained to detect the one or more objects of interest in the scene.

### Claim 11 (depends on 8)

The sensor device of claim 8, wherein the processor is further configured to retrieve, from a memory, respective calibration files specifying the respective conversion functions that reflect respective emissivity values of the one or more objects of interest detected in the scene, the respective calibration files being retrieved based on the identified respective surface characteristics of the one or more objects of interest detected in the scene.

### Claim 12 (depends on 11)

The sensor device of claim 11, wherein the processor is configured to
process the first image to detect at least i) a first object of interest in the scene, the first object of interest having first surface characteristics and ii) a second object of interest in the scene, the second object of interest having second surface characteristics different from the first surface characteristics,
retrieve, from the memory, a first calibration file specifying a first conversion function that is based on a first emissivity value corresponding to the first object of interest, and
retrieve, from the memory, a second calibration file specifying a second conversion function that reflects a second emissivity value of the second object of interest, the second emissivity value being different from the first emissivity value.

### Claim 13 (depends on 8)

The sensor device of claim 8, wherein the processor is further configured to cause the temperature values to be transmitted from the sensor device via a communication network to a server device for further processing.

### Claim 14 (depends on 8)

The sensor device of claim 8, wherein
the first camera is a visible light camera, and
the second camera is an infrared camera.

### Claim 15

A system, comprising
a server device application configured to process thermal data, and
a sensor device coupled to the server device component via a communication network, the sensor device comprising
a first camera configured to capture a first image of a scene,
a second camera configured to capture a second image of the scene, and
a processor coupled to the first camera and the second camera, the processor configured to
process the first image of the scene to detect one or more objects of interest in the scene, wherein detecting the one or more objects of interest includes identifying respective surface characteristics of the one or more objects of interest detected in the scene,
overlay the first image of the scene with the second image of the scene to identify one or more regions of interest in the second image of the scene corresponding to the one or more objects of interest detected in the first image of the scene,
convert values of pixels in the one or more regions of interest identified in the second image of the scene to temperature values, including converting values of pixels that belong to respective regions of interest, among the one or more regions of interest in the second image of the scene, using respective conversion functions that reflect respective emissivity values of the one or more objects of interest detected in the first image scene, the emissivity values determined by the identified respective surface characteristics of the one or more objects of interest detected in the first image of the scene, and
cause the temperature values to be transmitted via the communication network to the server device application for further processing.

### Claim 16 (depends on 15)

The system of claim 15, wherein the processor of the sensor device is to configured to process the first image of the scene using template matching to detect the one or more objects of interest in the scene, the processor being configured to match a template image of a particular object of interest in the scene across the first image to identify a matching region in the first image.

### Claim 17 (depends on 15)

The system of claim 15, wherein the processor of the sensor device is configured to configured to process the first image comprises using a neural network trained to detect the one or more objects of interest in the scene.

### Claim 18 (depends on 15)

The system of claim 15, wherein the processor of the sensor device is further configured to retrieve, from a memory, respective calibration files specifying the respective conversion functions that reflect respective emissivity values of the one or more objects of interest detected in the scene, the respective calibration files being retrieved based on the identified respective surface characteristics of the one or more objects of interest detected in the scene.

### Claim 19 (depends on 15)

The system of claim 15, wherein the sensor device is further configured to cause one or both of the first image and the second image to be transmitted via the network to the server device application.

### Claim 20 (depends on 15)

The system of claim 15, wherein the server device application is configured to cause one or more of i) the temperature values, ii) the first image and iii) the second image to be displayed to a user.
