# US 11,910,107 B2 · Image processing apparatus and image pickup apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,910,107 B2
- Document ID: US11910107B2
- Status: Granted
- Kind: B2
- Application number: 17/626,850
- Filed: 2020-06-01
- Published: 2022-08-18
- Granted: 2024-02-20
- Earliest claimed: 2019-07-24
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: Masatoshi Takashima
- Web page: https://patentgrep.com/patent/US11910107B2

## CPC

- H04N 25/11: Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N 25/135: {based on four or more different wavelength filter elements}
- G01J 3/0208: {using focussing or collimating elements, e.g. lenses or mirrors; performing aberration correction}
- G01J 3/0262: {Constructional arrangements for removing stray light}
- G01J 3/2803: {using photoelectric array detector}
- G01J 3/513: {having fixed filter-detector pairs}
- H04N 23/11: for generating image signals from visible and infrared light wavelengths
- H04N 23/843: {Demosaicing, e.g. interpolating colour pixel values}
- H04N 25/131: including elements passing infrared wavelengths
- G01J 2003/1213: {Filters in general, e.g. dichroic, band}
- G01J 2003/2806: {Array and filter array}
- G01J 2003/2826: {Multispectral imaging, e.g. filter imaging}
- G01J 2003/516: {with several stacked filters or stacked filter-detector pairs}
- H04N 23/85: for matrixing
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N)

## Abstract

Provided is an image pickup apparatus including a multi-band pass filter that selectively transmits band light in a specific band, a color filter that transmits the band light in the specific band per pixel of an image pickup device, the image pickup device that receives light transmitted by the multi-band pass filter and the color filter, and a signal processing section to which a pixel value of the image pickup device is input and which executes a signal process on the pixel value. The multi-band pass filter is configured to selectively transmit band light corresponding to a plurality of color filter elements, and the signal processing section generates a pixel value corresponding to a band, of a pixel to be processed using a raw image base pixel value signal per pixel and a spectral characteristic parameter of the image pickup apparatus.

## Claims

### Claim 1

An image pickup apparatus, comprising:
an image pickup device that includes a plurality of pixels;
a color filter that includes a plurality of color filter elements;
a multi-band pass filter configured to selectively transmit band light in a specific band, wherein
the band light corresponds to the plurality of color filter elements,
the color filter is configured to transmit the band light in the specific band to each of the plurality of pixels of the image pickup device, and
the image pickup device is configured to:
receive the band light transmitted by the multi-band pass filter and the color filter; and
generate a raw image based on the received band light; and
a signal processing section configured to:
receive the raw image from the image pickup device;
generate a plurality of raw image base pixel values based on the raw image received from the image pickup device; and
generate a true pixel value, corresponding to the specific band, of each of the plurality of pixels based on the generated plurality of raw image base pixel values and a spectral characteristic parameter of the image pickup apparatus, wherein
the spectral characteristic parameter of the image pickup apparatus represents an intensity ratio of a signal transmitted by a first color filter element of the plurality of color filter elements to a signal transmitted by a second color filter element of the plurality of color filter elements in the specific band of the multi-band pass filter.

### Claim 2 (depends on 1)

The image pickup apparatus according to claim 1, wherein the signal processing section is further configured to:
execute iterative calculation of an equation of calculating the true pixel value corresponding to the specific band using the plurality of raw image base pixel values and the spectral characteristic parameter, and
calculate a convergence value by the iterative calculation as the true pixel value corresponding to the specific band.

### Claim 3 (depends on 1)

The image pickup apparatus according to claim 1, wherein the specific band includes a transmitted light band of the multi-band pass filter.

### Claim 4 (depends on 1)

The image pickup apparatus according to claim 1, wherein the signal processing section is further configured to generate the plurality of raw image base pixel values by a demosaicing process on the raw image received from the image pickup device.

### Claim 5 (depends on 1)

The image pickup apparatus according to claim 1, wherein the signal processing section is further configured to:
generate the plurality of raw image base pixel values in units of a plurality of pixel blocks of the raw image received from the image pickup device, and
calculate the true pixel value corresponding to the specific band in units of the plurality of pixel blocks.

### Claim 6 (depends on 1)

The image pickup apparatus according to claim 1, wherein
the color filter includes an RGB color filter,
the multi-band pass filter is further configured to selectively transmit the band light of light in RGB bands, and
the signal processing section is further configured to:
execute iterative calculation of an equation of calculating the true pixel value corresponding to the specific band, and
calculate a convergence value by the iterative calculation as each of RGB pixel values of each of the plurality of pixels.

### Claim 7 (depends on 6)

The image pickup apparatus according to claim 6, wherein the signal processing section is further configured to generate a noise-reduced high precision color image based on the RGB pixel values.

### Claim 8 (depends on 1)

The image pickup apparatus according to claim 1, wherein
the color filter includes an RGBIR color filter,
the multi-band pass filter is further configured to selectively transmit each band light of light in RGBIR bands, and
the signal processing section is further configured to:
execute iterative calculation of an equation of calculating the true pixel value corresponding to the specific band, and
calculate a convergence value by the iterative calculation as each of RGBIR pixel values of each of the plurality of pixels.

### Claim 9 (depends on 8)

The image pickup apparatus according to claim 8, wherein the signal processing section is further configured to calculate, based on the RGBIR pixel values, an activity index value of vegetation that is a photographic subject.

### Claim 10

An image processing apparatus, comprising:
a signal processing section configured to:
receive a raw image from an image pickup device of a multispectral camera, wherein
the raw image is based on band light in a specific band transmitted by each of a multi-band pass filter and a plurality of color filter elements of the multispectral camera, and
the band light is transmitted by the plurality of color filter elements to each of a plurality of pixels of the image pickup device;
generate a plurality of raw image base pixel values based on the raw image; and
generate a true pixel value, corresponding to the specific band, of each of the plurality of pixels based on the generated plurality of raw image base pixel values and a spectral characteristic parameter of the multispectral camera, wherein
the spectral characteristic parameter of the multispectral camera is an intensity ratio of a signal transmitted by a first color filter element of the plurality of color filter elements to a signal transmitted by a second color filter element of the plurality of color filter elements in the specific band of the multi-band pass filter.

### Claim 11 (depends on 10)

The image processing apparatus according to claim 10, wherein the signal processing section is further configured to:
execute iterative calculation of an equation of calculating the true pixel value corresponding to the specific band based on the plurality of raw image base pixel values and the spectral characteristic parameter, and
calculate a convergence value by the iterative calculation as the true pixel value corresponding to the specific band.

### Claim 12 (depends on 10)

The image processing apparatus according to claim 10, wherein the specific band includes a transmitted light band of the multi-band pass filter of the multispectral camera.

### Claim 13 (depends on 10)

The image processing apparatus according to claim 10, wherein the signal processing section is further configured to:
execute iterative calculation of an equation of calculating the true pixel value corresponding to the specific band,
calculate each of RGB pixel values of each of the plurality of pixels to be processed by the iterative calculation, and
generate a noise-reduced high precision color image based on the RGB pixel values.

### Claim 14 (depends on 10)

The image processing apparatus according to claim 10, wherein the signal processing section is further configured to:
execute iterative calculation of an equation of calculating the true pixel value corresponding to the specific band,
calculate each of RGBIR pixel values of each of the plurality of pixels to be processed by the iterative calculation, and
calculate, based on the RGBIR pixel values, an activity index value of vegetation that is a photographic subject of the multispectral camera.

### Claim 15

An image processing method executed by an image pickup apparatus, the image processing method comprising:
selectively transmitting, by a multi-band pass filter of the image pickup apparatus, band light in a specific band, wherein the band light corresponds to a plurality of color filter elements of a color filter of the image pickup apparatus;
transmitting, by the color filter, the band light in the specific band to each of a plurality of pixels of an image pickup device of the image pickup apparatus;
receiving, by the image pickup device, the band light transmitted by the multi-band pass filter and the color filter;
generating, by the image pickup device, a raw image based on the received band light;
generating, by a signal processing section of the image pickup apparatus, a plurality of raw image base pixel values based on the raw image from the image pickup device; and
generating, by the signal processing section, a true pixel value, corresponding to the specific band, of each of the plurality of pixels based on the generated plurality of raw image base pixel values and a spectral characteristic parameter of the image pickup apparatus, wherein
the spectral characteristic parameter of the image pickup apparatus is an intensity ratio of a signal transmitted by a first color filter element of the plurality of color filter elements to a signal transmitted by a second color filter element of the plurality of color filter elements in the specific band of the multi-band pass filter.

### Claim 16

An image processing method executed by an image processing apparatus, the image processing method comprising:
receiving a raw image from an image pickup device of a multispectral camera, wherein
the raw image is based on band light in a specific band transmitted by each of a multi-band pass filter and a plurality of color filter elements of the multispectral camera, and
the band light is transmitted by the plurality of color filter elements to each of a plurality of pixels of the image pickup device;
generating a plurality of raw image base pixel values based on the raw image; and
generating a true pixel value, corresponding to the specific band, of each of the plurality of pixels based on the generated plurality of raw image base pixel values and a spectral characteristic parameter of the multispectral camera, wherein
the spectral characteristic parameter of the multispectral camera is an intensity ratio of a signal transmitted by a first color filter element of the plurality of color filter elements to a signal transmitted by a second color filter element of the plurality of color filter elements in the specific band of the multi-band pass filter.

### Claim 17

A non-transitory computer-readable medium having stored thereon, computer-executable instructions which, when executed by an image processing apparatus, cause the image processing apparatus to execute operations, the operations comprising:
receiving a raw image from an image pickup device of a multispectral camera, wherein
the raw image is based on band light in a specific band transmitted by each of a multi-band pass filter and a plurality of color filter elements of the multispectral camera, and
the band light is transmitted by the plurality of color filter elements to each of a plurality of pixels of the image pickup device;
generating a plurality of raw image base pixel values based on the raw image; and
generating a true pixel value, corresponding to the specific band, of each of the plurality of pixels based on the generated plurality of raw image base pixel values and a spectral characteristic parameter of the multispectral camera, wherein
the spectral characteristic parameter of the multispectral camera is an intensity ratio of a signal transmitted by a first color filter element of the plurality of color filter elements to a signal transmitted by a second color filter element of the plurality of color filter elements in the specific band of the multi-band pass filter.