# US 2021/0281713 A1 · Image processing device, image processing method, image sensor, and non-transitory computer readable recording medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0281713 A1
- Document ID: US20210281713A1
- Status: Published application
- Kind: A1
- Application number: 17/053,788
- Filed: 2019-05-22
- Published: 2021-09-09
- Earliest claimed: 2018-06-12
- Assignees: OMRON Corporation
- Applicants: OMRON Corporation
- Inventors: Yutaka KATO
- Web page: https://patentgrep.com/patent/US20210281713A1

## CPC

- H04N 1/60: Colour correction or control {(H04N1/54 takes precedence)}
- G06F 18/2431: Multiple classes
- G06T 7/90: Determination of colour characteristics
- H04N 1/54: Conversion of colour picture signals to a plurality of signals some of which represent particular mixed colours, e.g. for textile printing
- H04N 1/603: {controlled by characteristics of the picture signal generator or the picture reproducer}
- H04N 23/10: for generating image signals from different wavelengths
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Provided is an image processing device which has: an image input part configured to input a multi-spectral image which includes N (N is an integer of 4 or more) channels corresponding to N bands; a reference vector setting part configured to cause a user to set, as three reference vectors, three types of N-dimensional vectors having sensitivities of the respective N bands as elements; and a conversion part configured to convert the multi-spectral image into a 3-channel image including three channels by decomposing a spectral distribution of each pixel of the multi-spectral image on the basis of the three reference vectors.

## Claims

### Claim 1

An image processing device comprising:
an image input part configured to input a multi-spectral image which comprises N (N is an integer of 4 or more) channels corresponding to N bands;
a reference vector setting part configured to cause a user to set, as three reference vectors, three types of N-dimensional vectors having sensitivities of the respective N bands as elements; and
a conversion part configured to convert the multi-spectral image into a 3-channel image comprising three channels by decomposing a spectral distribution of each pixel of the multi-spectral image on the basis of the three reference vectors.

### Claim 2 (depends on 1)

The image processing device according to claim 1,
wherein the reference vector setting part displays the reference vectors by a sensitivity curve representing sensitivities at each wavelength in a graph representing wavelength on the horizontal axis and sensitivity on the vertical axis, and causes a user to input parameters that define the shape of the sensitivity curve.

### Claim 3 (depends on 2)

The image processing device according to claim 2,
wherein the reference vector setting part displays the sensitivity curve as a symmetrical distribution curve, and
the parameters that define the shape of the sensitivity curve comprise a mean of distribution, a height of distribution, and spread of distribution.

### Claim 4 (depends on 2)

The image processing device according to claim 2,
wherein the reference vector setting part superimposes and displays three reference vectors on a screen that displays spectral distribution of a first pixel and spectral distribution of a second pixel in a comparative manner.

### Claim 5 (depends on 4)

The image processing device according to claim 4,
wherein the reference vector setting part extracts three wavelengths having a large difference between the spectral distribution of the first pixel and the spectral distribution of the second pixel, creates three reference vector candidates according to the extracted three wavelengths, and recommends the candidates to the user.

### Claim 6 (depends on 4)

The image processing device according to claim 4,
wherein the reference vector setting part creates three reference vector candidates so that the difference between a value of the converted pixel obtained by decomposing the spectral distribution of the first pixel on the basis of the three reference vectors and a value of the converted pixel obtained by decomposing the spectral distribution of the second pixel on the basis of the three reference vectors is maximized, and recommends the candidates to the user.

### Claim 7 (depends on 1)

The image processing device according to claim 1,
wherein the reference vector setting part automatically changes, in accordance with change of any of the three reference vectors by the user, the reference vectors other than the changed reference vector so that the three reference vectors satisfy a predetermined constraint.

### Claim 8 (depends on 7)

The image processing device according to claim 7,
wherein the predetermined constraint is a constraint that the three reference vectors are orthogonal to each other.

### Claim 9

An image processing device comprising:
an image input part configured to input a multi-spectral image which comprises N (N is an integer of 4 or more) channels corresponding to N bands;
a reference vector setting part configured to set, as three reference vectors, three types of N-dimensional vectors having sensitivities of the respective N bands as elements; and
a conversion part configured to convert the multi-spectral image into a 3-channel image comprising three channels by decomposing a spectral distribution of each pixel of the multi-spectral image on the basis of the three reference vectors,
wherein the reference vector setting part extracts a plurality of sample pixels from one or more multi-spectral images, and obtains the three reference vectors by statistical processing based on the distribution of the plurality of sample pixels in an N-dimensional space.

### Claim 10 (depends on 9)

The image processing device according to claim 9,
wherein the reference vector setting part obtains a first principal component, a second principal component, and a third principal component by principal component analysis, based on the distribution of the plurality of sample pixels in the N-dimensional space, and sets the first to third principal components as the three reference vectors.

### Claim 11 (depends on 9)

The image processing device according to claim 9,
wherein when the plurality of sample pixels are pixel groups to be classified into two categories,
the reference vector setting part obtains a discrimination boundary of the two categories by discriminant analysis based on the distribution of the plurality of sample pixels in the N-dimensional space, determines one reference vector so as to make it orthogonal to the discrimination boundary, and determines the other two reference vectors so as to make them orthogonal to the one reference vector.

### Claim 12 (depends on 9)

The image processing device according to claim 9,
wherein when the plurality of sample pixels are pixel groups to be classified into three categories,
the reference vector setting part obtains a discrimination boundary of a first category and a second category, a discrimination boundary of the second category and a third category, and a discrimination boundary of the third category and the first category respectively by discriminant analysis, based on the distribution of the plurality of sample pixels in the N-dimensional space, and determines the three reference vectors so as to make them orthogonal to the three discrimination boundaries respectively.

### Claim 13 (depends on 1)

The image processing device according to claim 1,
comprising an output part to output the 3-channel image converted by the conversion part as data in the form of an RGB image.

### Claim 14 (depends on 13)

The image processing device according to claim 13,
wherein among three channels of the 3-channel image, the output part allocates the channel corresponding to the longest wavelength to an R channel, the channel corresponding to the second longest wavelength to a G channel, and the channel corresponding to the shortest wavelength to a B channel, respectively.

### Claim 15 (depends on 1)

The image processing device according to claim 1,
wherein the conversion part obtains the value of each pixel of the 3-channel image by calculating the inner product of the spectral distribution of each pixel of the multi-spectral image and each reference vector.

### Claim 16 (depends on 1)

An image sensor comprising:
a multispectral camera capable of outputting a multi-spectral image which comprises N (N is an integer of 4 or more) channels corresponding to N bands, and
the image processing device according to claim 1, which converts the multi-spectral image output from the multispectral camera into a 3-channel image.

### Claim 17

An image processing method comprising:
a step of inputting a multi-spectral image which comprises N (N is an integer of 4 or more) channels corresponding to N bands,
a step of causing a user to set, as three reference vectors, three types of N-dimensional vectors having sensitivities of the respective N bands as elements, and
a step of converting the multi-spectral image into a 3-channel image comprising three channels by decomposing a spectral distribution of each pixel of the multi-spectral image on the basis of the three reference vectors.

### Claim 18

An image processing method comprising:
a step of inputting a multi-spectral image which comprises N (N is an integer of 4 or more) channels corresponding to N bands,
a step of setting, as three reference vectors, three types of N-dimensional vectors having sensitivities of the respective N bands as elements, and
a step of converting the multi-spectral image into a 3-channel image comprising three channels by decomposing a spectral distribution of each pixel of the multi-spectral image on the basis of the three reference vectors,
wherein in the step of setting the reference vectors, a plurality of sample pixels are extracted from one or more multi-spectral images, and the three reference vectors are obtained by statistical processing based on the distribution of the plurality of sample pixels in an N-dimensional space.

### Claim 19 (depends on 17)

A non-transitory computer readable recording medium storing a program for causing a processor to execute each step of the image processing method according to claim 17.
