# US 2019/0378257 A1 · Multi-spectrum-based image fusion apparatus and method, and image sensor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378257 A1
- Document ID: US20190378257A1
- Status: Published application
- Kind: A1
- Application number: 16/480,270
- Filed: 2018-01-25
- Published: 2019-12-12
- Earliest claimed: 2017-02-10
- Assignees: Hangzhou Hikvision Digital Technology Co., Ltd.
- Applicants: Hangzhou Hikvision Digital Technology Co., Ltd.
- Inventors: Meng FAN; Hai YU; Shiliang PU
- Web page: https://patentgrep.com/patent/US20190378257A1

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- H04N 23/73: by influencing the exposure time
- H04N 9/646: {for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters}
- H04N 9/67: for matrixing (camera processing pipelines for matrixing of colour signals H04N23/85)
- H04N 9/78: for separating the brightness signal or the chrominance signal from the colour television signal, e.g. using comb filter
- H04N 23/10: for generating image signals from different wavelengths
- H04N 23/56: provided with illuminating means
- H04N 23/71: Circuitry for evaluating the brightness variation
- H04N 23/72: Combination of two or more compensation controls
- H04N 25/131: including elements passing infrared wavelengths
- H04N 25/133: including elements passing panchromatic light, e.g. filters passing white light
- H04N 25/135: {based on four or more different wavelength filter elements}
- G06T 2207/10024: Color image
- G06T 2207/10048: Infrared image
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A multi-spectrum based image fusion apparatus is disclosed, which includes a light acquisition device, an image processor, and an image sensor having five types of photosensitive channels. The five types of photosensitive channels including red, green and blue RGB channels, an infrared IR channel and a full-band W channel. The light acquisition device acquires target light corresponding to incident light. The image sensor converts the target light into an image signal through the RGB channels, the IR channel and the W channel. The image processor analyzes the image signal into RGB color signals and a brightness signal, and fuses the RGB color signals and the brightness signal to obtain a fused image. The collection of the channels based on which the RGB color signals and the brightness signal are obtained includes the five types of photosensitive channels.

## Claims

### Claim 1

A multi-spectrum based image fusion apparatus, comprising:
a light acquisition device, an image processor, and an image sensor having five types of photosensitive channels, the five types of photosensitive channels comprising red, green and blue RGB channels, an infrared IR channel and a full-band W channel; wherein
the light acquisition device is configured to acquire target light corresponding to incident light;
the image sensor is configured to convert the target light into an image signal through the RGB channels, the IR channel and the W channel; and
the image processor is configured to analyze the image signal into RGB color signals and a brightness signal, and fuse the RGB color signals and the brightness signal to obtain a fused image, wherein a collection of the channels based on which the RGB color signals and the brightness signal are obtained comprises the five types of photosensitive channels.

### Claim 2 (depends on 1)

The multi-spectrum based image fusion apparatus of claim 1, wherein the process that the image processor analyzes the image signal into the RGB color signals and the brightness signal comprises:
performing an interpolation operation on the W channel of the image signal to generate a brightness signal that has a resolution same as an input resolution, wherein the input resolution is a resolution of the image signal;
performing interpolation operations on the RGB channels and IR channel of the image signal to generate an image signal of each of the channels that has a resolution same as the input resolution; and
traversing the image signal of each of the channels, and subtracting a value of an IR parameter for a pixel position respectively from values of the traversed R channel, G channel, and B channel to generate the RGB color signals, wherein the value of the IR parameter is a product of a value of the IR channel at the pixel position and a preset correction value,
or
wherein the process that the image processor analyzes the image signal into the RGB color signals and the brightness signal comprises:
performing interpolation operations on the RGB channels, the IR channel and the W channel of the image signal to generate an image signal of each of the channels that has a resolution same as an input resolution;
generating the brightness signal by using the image signal of each of the channels, a first predetermined formula and a second predetermined formula; wherein the first predetermined formula is y=x1*R+x2*G+x3*B, and x1, x2, and x3 are weights, and wherein the second predetermined formula is the brightness signal=x4*W+x5*IR+x6*y, and x4, x5, and x6 are weights;
traversing the image signal of each of the channels, and subtracting a value of an IR parameter for a pixel position respectively from values of the traversed R channel, G channel, and B channel to generate first standby RGB color signals, wherein the value of the IR parameter is a product of a value of the IR channel at the pixel position and a preset correction value;
traversing the image signal of each of the channels, and subtracting a value of an IR′ for a pixel position respectively from values of the traversed R channel, G channel, and B channel to generate second standby RGB color signals, wherein the value of the IR′ for the pixel position is obtained by a third predetermined formula as IR′=(R+G+B−W)/n; and
weighting and combining the first standby RGB color signals and the second standby RGB color signals to generate the RGB color signals.

### Claim 3

(canceled)

### Claim 4 (depends on 1)

The multi-spectrum based image fusion apparatus of claim 1, wherein the process that the image processor fuses the RGB color signals and the brightness signal to obtain the fused image comprises:
converting the RGB color signals into YUV signals, wherein the YUV signals are luminance and chrominance signals;
extracting chrominance UV components in the YUV signals;
combining the UV components with the brightness signal to form new YUV signals; and
determining the new YUV signals as the fused image, or converting the new YUV signals into RGB signals and determining the resulting RGB signals as the fused image,
or
wherein the process that the image processor fuses the RGB color signals and the brightness signal to obtain the fused image comprises:
calculating an auxiliary value Y for each pixel according to the RGB color signals, as Y=(R*w1+B*w2+G*w3)/(w1+w2+w3), wherein R is a value of the R channel corresponding to the pixel, G is a value of the G channel corresponding to the pixel, B is a value of the B channel corresponding to the pixel, and w1, w2 and w3 are weights;
calculating a ratio of the value of each channel of the RGB color signals to the auxiliary value Y to obtain reference channel values K1, K2, and K3 for the pixel, which are calculated as K1=R/Y, K2=G/Y, K3=B/Y;
performing chroma noise reduction on the reference channel values K1, K2, and K3; and
fusing the brightness signal Y′ with the reference channel values K1-K3 subjected to the chroma noise reduction at the pixel, to generate fused RGB channel values R′, G′ and B′ and obtain the fused image; which are calculated as R′=K1*Y′; G′=K2*Y′; and B′=K3*Y′.

### Claim 5

(canceled)

### Claim 6 (depends on 1)

The multi-spectrum based image fusion apparatus of claim 1, wherein the light acquisition device is further configured to filter a spectrum component of a first predetermined wavelength range from the incident light to obtain the target light; wherein the first predetermined wavelength range is a spectrum wavelength range in which a difference between responsivities of the RGB channels and of a brightness channel of the image sensor at an infrared band is greater than a first predetermined threshold, the brightness channels being the IR channel or W channel.

### Claim 7 (depends on 6)

The multi-spectrum based image fusion apparatus of claim 6,
wherein the light acquisition device comprises a band-stop filter and an optical lens of a first type; wherein
the optical lens of the first type is configured to transmit the incident light to the band-stop filter in a full spectrum transmission manner; and
the band-stop filter is configured to filter the spectrum component of the first predetermined wavelength range from the light transmitted from the optical lens of the first type to obtain the target light,
or
wherein the light acquisition device comprises an optical lens of a second type, which is capable of filtering the spectrum component of the first predetermined wavelength range.

### Claim 8

(canceled)

### Claim 9 (depends on 6)

The multi-spectrum based image fusion apparatus of claim 6, wherein the first predetermined wavelength range is [T1, T2], wherein the value of T1 is in a range of [600 nm, 800 nm], and the value of T2 is in a range of [750 nm, 1100 nm].

### Claim 10 (depends on 1)

The multi-spectrum based image fusion apparatus of claim 1, wherein during the conversion from the target light to the image signal by the image sensor, in a second predetermined wavelength range of the infrared band, differences between responsivities of the RGB channels are lower than a second predetermined threshold.

### Claim 11 (depends on 10)

The multi-spectrum based image fusion apparatus of claim 10, wherein the second predetermined wavelength range is [T3, T4], wherein T4 is greater than T3, T3 is greater than or equal to 750 nm, and T4 is less than or equal to 1100 nm.

### Claim 12 (depends on 1)

The multi-spectrum based image fusion apparatus of claim 1, wherein during the conversion from the target light into the image signal, the image sensor performs a multiple-exposures acquisition within one frame time.

### Claim 13 (depends on 12)

The multi-spectrum based image fusion apparatus of claim 12, wherein the process that the image processor analyzes the image signal into the RGB color signals and the brightness signal comprises:
analyzing an image signal generated by using an exposure of a first type to obtain the RGB color signals; and
analyzing an image signal generated by using an exposure of a second type to obtain the brightness signal.

### Claim 14 (depends on 13)

The multi-spectrum based image fusion apparatus of claim 13, wherein an exposure duration of the exposure of the first type is less than that of the exposure of the second type.

### Claim 15 (depends on 1)

The multi-spectrum based image fusion apparatus of claim 1, further comprising a signal controller, configured to control the image sensor to generate an image signal that meets a predetermined brightness requirement.

### Claim 16 (depends on 15)

The multi-spectrum based image fusion apparatus of claim 15, wherein the signal controller is further configured to analyze brightness of image signals generated by the image sensor, and control the image sensor according to an analyzing result to generate the image signal that meets the predetermined brightness requirement,
or
the signal controller is further configured to control the image sensor to switch between one-exposure acquisition and multiple-exposures acquisition.

### Claim 17

(canceled)

### Claim 18 (depends on 15)

The multi-spectrum based image fusion apparatus of claim 15, further comprising an infrared light filling device; wherein the signal controller is further configured to control the infrared light filling device to fill infrared light for the image sensor.

### Claim 19

A multi-spectrum based image fusion method, applicable to a multi-spectrum based image fusion apparatus having five types of photosensitive channels, wherein the five types of photosensitive channels comprises red, green and blue RGB channels, an infrared IR channel and a full-band W channel; the method comprising:
acquiring target light corresponding to incident light;
converting the target light into an image signal through the RGB channels, the IR channel and the W channel; and
analyzing the image signal into RGB color signals and a brightness signal, and fusing the RGB color signals and the brightness signal to obtain a fused image, wherein the collection of the channels based on which the RGB color signals and the brightness signal are obtained comprises the five types of photosensitive channels.

### Claim 20 (depends on 19)

The method of claim 19, wherein the step of analyzing the image signal into the RGB color signals and the brightness signal comprises:
performing an interpolation operation on the W channel of the image signal to generate a brightness signal that has a resolution same as an input resolution, wherein the input resolution is a resolution of the image signal;
performing interpolation operations on the RGB channels and IR channel of the image signal to generate an image signal of each of the channels that has a resolution same as the input resolution; and
traversing the image signal of each of the channels, and subtracting a value of an IR parameter for a pixel position respectively from values of the traversed R channel, G channel, or B channel to generate the RGB color signals, wherein the value of the IR parameter is a product of a value of the IR channel at the pixel position and a preset correction value,
or
wherein the step of analyzing the image signal into the RGB color signals and the brightness signal comprises:
performing interpolation operations on the RGB channels, the IR channel and the W channel of the image signal to generate an image signal of each of the channels that has a resolution same as an input resolution;
generating the brightness signal by using the image signal of each of the channels, a first predetermined formula and a second predetermined formula; wherein the first predetermined formula is y=x1*R+x2*G+x3*B, and x1, x2, and x3 are weights, and wherein the second predetermined formula is the brightness signal=x4*W+x5*IR+x6*y, and x4, x5, and x6 are weights;
traversing the image signal of each of the channels, and subtracting a value of an IR parameter for a pixel position respectively from values of the traversed R channel, G channel, and B channel to generate first standby RGB color signals, where the value of the IR parameter is a product of a value of the IR channel at the pixel position and a preset correction value;
traversing the image signal of each of the channels, and subtracting a value of an IR′ for a pixel position respectively from values of the traversed R channel, G channel, and B channel to generate second standby RGB color signals, where a value of IR′ for the pixel position is obtained by a third predetermined formula as IR′=(R+G+B−W)/n; and
weighting and combining the first standby RGB color signals and the second standby RGB color signals to generate the RGB color signals,
or
wherein the step of fusing the RGB color signals and the brightness signal to obtain the fused image comprises:
converting the RGB color signals into YUV signals, wherein the YUV signals are luminance and chrominance signals;
extracting chrominance UV components in the YUV signals;
combining the UV components with the brightness signal to form new YUV signals; and
determining the new YUV signals as the fused image, or converting the new YUV signals into RGB signals and determining the resulting RGB signals as the fused image,
or
wherein the step of fusing the RGB color signals and the brightness signal to obtain a fused image comprises:
calculating an auxiliary value Y for each pixel according to the RGB color signals, as Y=(R*w1+B*w2+G*w3)/(w1+w2+w3), wherein R is a value of the R channel corresponding to the pixel, G is a value of the G channel corresponding to the pixel, B is a value of the B channel corresponding to the pixel, and w1, w2 and w3 are weights;
calculating a ratio of the value of each channel of the RGB color signals to the auxiliary value Y to obtain reference channel values K1, K2, and K3 for each pixel, which are calculated as K1=R/Y, K2=G/Y, K3=B/Y;
performing chroma noise reduction on the reference channel values K1, K2, and K3; and
fusing the brightness signal Y′ with the reference channel values K1-K3 subjected to the chroma noise reduction at the pixel, to generate fused RGB channel values R′, G′ and B′ and to obtain the fused image; wherein R′, G′ and B′ are calculated as R′=K1*Y′; G′=K2*Y′;
and B′=K3*Y′.

### Claim at position 21

21-23. (canceled)

### Claim 24

An image sensor, having five types of photosensitive channels, wherein the five types of photosensitive channels comprises red, green and blue RGB channels, an infrared IR channel and a full-band W channel, a proportion of pixels corresponding to the W channel is greater than a proportion of pixels corresponding to any other photosensitive channels.

### Claim 25 (depends on 24)

The image sensor of claim 24, wherein the proportion of pixels corresponding to the W channel is ½, and wherein
the proportion pixels corresponding to each photosensitive channel of the RGB channels and the IR channel is the same,
or
the proportion of pixels corresponding to at least one photosensitive channel of the RGB channels and the IR channel is different from the proportions of pixels corresponding to the other photosensitive channels.

### Claim at position 24

26-27. (canceled)

### Claim 28 (depends on 25)

The image sensor of claim 25, wherein the proportion of pixels corresponding to each photosensitive channel of the RGB channels is the same, and wherein
when the proportion of pixels corresponding to the W channel is ½, pixels corresponding to each of the five types of photosensitive channels are arranged as a uniform mesh.

### Claim 29

(canceled)

### Claim 30 (depends on 19)

A non-transitory storage medium for storing executable codes that, when executed, perform the multi-spectrum based image fusion method of claim 19.

### Claim 31

(canceled)
