# US 2025/0168307 A1 · Apparatus for and method for image processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0168307 A1
- Document ID: US20250168307A1
- Status: Published application
- Kind: A1
- Application number: 18/800,432
- Filed: 2024-08-12
- Published: 2025-05-22
- Earliest claimed: 2023-11-17
- Assignees: Samsung Electronics Co., Ltd.; Ewha University-Industry Collaboration Foundation
- Applicants: Samsung Electronics Co., Ltd.; Ewha University - Industry Collaboration Foundation
- Inventors: Woo-Shik KIM; Hyejin OH; Jewon KANG; Sangyoon LEE; Heejin CHOI; Yungkyung PARK
- Web page: https://patentgrep.com/patent/US20250168307A1

## CPC

- H04N 9/73: Colour balance circuits, e.g. white balance circuits or colour temperature control (camera processing pipelines for colour balance H04N23/88)
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06V 10/82: using neural networks
- H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus for obtaining an image may include a multispectral image sensor configured to obtain an image through four or more channels, and a processor configured to estimate illumination information by inputting the obtained image to a pre-trained deep learning network and perform color transformation for the obtained image based on the estimated illumination information, wherein the deep learning network learns a channel correlation between the channels to output the estimated illumination information.

## Claims

### Claim 1

An apparatus for image processing, the apparatus comprising:
a multispectral image sensor configured to obtain an image through a plurality of channels; and
a processor configured to estimate illumination information by inputting the obtained image to a neural network and perform color transformation on the obtained image based on the estimated illumination information,
wherein the neural network is trained using a channel correlation between the plurality of channels to output the estimated illumination information.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the processor is further configured to input the channel correlation to at least one of a plurality of layers constituting the neural network.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the channel correlation comprises at least one of a first channel correlation between first channels and a second channel correlation between second channels, the first channels corresponding to the obtained image, and the second channels corresponding to an intermediate image output from one of the plurality of layers.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the neural network comprises:
a convolution block comprising a plurality of convolution layers; and
a channel attention block following at least one of the plurality of convolution layers,
wherein the channel attention block comprises a pooling layer and a first fully connected layer.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the processor is further configured to generate a first channel vector by performing global pooling on the image and add the first channel vector to an output result of the first fully connected layer.

### Claim 6 (depends on 4)

The apparatus of claim 4, wherein the processor is further configured to generate a second channel vector by performing global pooling on an intermediate image output from a first convolution layer among the plurality of convolution layers,
the channel attention block follows a second convolution layer following the first convolution layer, and
the second channel vector is added to an output result of the first fully connected layer.

### Claim 7 (depends on 4)

The apparatus of claim 4, wherein the channel attention block is configured to apply either one or both of a rectified linear unit (ReLU) function and a sigmoid function.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the neural network comprise at least one self-attention layer configured to learn the channel correlation.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the neural network is pre-trained by using a loss function including a first angular error and a second angular error,
the first angular error is based on first ground truth illumination information and first estimation illumination information regarding a training image for training the neural network, and
the second angular error is based on second illumination information and second estimation illumination information that are obtained by transforming the first ground truth illumination information and the first estimation illumination information into an XYZ color space by using color matching function, respectively.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein, when the obtained image is a mixed illumination image obtained from an environment where a first illumination source and a second illumination source are present, the processor is further configured to divide the mixed illumination image into divided regions, estimate first illumination information based on a first region where the first illumination source is dominant among the divided regions, estimate second illumination information based on a second region where the second illumination source is dominant among the divided regions, and estimate mixed illumination information regarding the mixed illumination image based on the first illumination information and the second illumination information.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein the processor is further configured to adjust a dimension of an illumination vector corresponding to the estimated illumination information to be greater than a number of channels corresponding to the obtained image or adjust the dimension of the illumination vector to 3.

### Claim 12 (depends on 1)

The apparatus of claim 1, wherein the neural network comprises:
a three-dimensional (3D) convolution layer configured to perform 3D convolution based on the plurality of channels and a two-dimensional (2D) image corresponding to the obtained image; and
a spectral channel attention block following the 3D convolution layer.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the neural network further comprises a spectral self-attention layer configured to learn the channel correlation.

### Claim 14

A method for image processing, the method comprising:
obtaining an image through a plurality of channels by using a multispectral image sensor;
estimating illumination information by inputting the obtained image to a neural network; and
based on the estimated illumination information, performing color transformation for the obtained image,
wherein the neural network is trained using a channel correlation between the plurality of channels to output the estimated illumination information.

### Claim 15 (depends on 14)

The method of claim 14, wherein the estimating of the illumination information further comprises:
inputting the obtained image to the neural network; and
inputting the channel correlation to at least one of a plurality of layers constituting the neural network.

### Claim 16 (depends on 15)

The method of claim 15, wherein the channel correlation comprises at least one of a first channel correlation between first channels and a second channel correlation between second channels, the first channels corresponding to the obtained image, and the second channels corresponding to an intermediate image output from at least one of the plurality of layers.

### Claim 17 (depends on 14)

The method of claim 14, wherein the neural network comprises:
a convolution block comprising a plurality of convolution layers; and
a channel attention block following at least one of the plurality of convolution layers,
wherein the channel attention block comprises a pooling layer and a first fully connected layer.

### Claim 18 (depends on 17)

The method of claim 17, wherein the estimating of the illumination information further comprises:
generating a first channel vector by performing global pooling on the obtained image; and
adding the first channel vector to an output result of the first fully connected layer.

### Claim 19 (depends on 17)

The method of claim 17, wherein the estimating of the illumination information further comprises:
generating a second channel vector by performing global pooling on an intermediate image output from a first convolution layer among the plurality of convolution layers; and
adding the second channel vector to an output result of the first fully connected layer,
wherein the channel attention block follows a second convolution layer following the first convolution layer.

### Claim 20 (depends on 14)

The method of claim 14, wherein the estimating of the illumination information further comprises adjusting a dimension of an illumination vector corresponding to the estimated illumination information to be greater than a number of channels corresponding to the obtained image or adjusting the dimension of the illumination vector to 3.
