# US 2023/0410463 A1 · Method for Obtaining Light Source Spectrum and Device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410463 A1
- Document ID: US20230410463A1
- Status: Published application
- Kind: A1
- Application number: 18/037,723
- Filed: 2021-11-19
- Published: 2023-12-21
- Earliest claimed: 2020-11-23
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Maosen Zhou; Yanlin Qian; Minggui He; Kang Qian; Yongxing Yang; Miaofeng Wang
- Web page: https://patentgrep.com/patent/US20230410463A1

## CPC

- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 7/90: Determination of colour characteristics
- G06V 10/143: Sensing or illuminating at different wavelengths
- G06V 10/56: relating to colour
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/762: using clustering, e.g. of similar faces in social networks
- G06V 10/763: {Non-hierarchical techniques, e.g. based on statistics of modelling distributions}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- H04M 1/72439: for image or video messaging
- H04N 9/31: Projection devices for colour picture display {, e.g. using electronic spatial light modulators [ESLM]}
- H04N 23/16: Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction
- H04N 23/70: Circuitry for compensating brightness variation in the scene
- H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control
- G06T 2207/10024: Color image
- G06T 2207/10036: Multispectral image; Hyperspectral image
- G06T 2207/20072: Graph-based image processing
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for obtaining a light source spectrum includes: obtaining first information in a current photographing scene, where the first information includes at least one of a first image generated by a red, green, and blue (RGB) sensor or a light intensity of light received by each pixel on a first multispectral sensor; inputting the first information into a first model to obtain a probability that a light source in the current photographing scene belongs to each type of light source; and determining a spectrum of the light source in the current photographing scene based on the probability that the light source in the current photographing scene belongs to each type of light source and a spectrum of each type of light source.

## Claims

### Claim 1

A method for obtaining a light source spectrum comprising:
obtaining first information in a current photographing scene, wherein the first information comprises at least one of a first image generated by a red, green, and blue (RGB) sensor or a light intensity of a light received by each pixel on a first multispectral sensor;
inputting the first information into a first model, to obtain a probability that a first light source in the current photographing scene belongs to each type of light source; and
determining a first spectrum of the first light source based on the probability and a second spectrum of each type of light source.

### Claim 2 (depends on 1)

The method of claim 1, wherein before inputting the first information into the first model, the method further comprises:
obtaining a training sample comprising second information in different photographing scenes and light source types corresponding to light sources in the different photographing scenes; and
training the first model using the training sample.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
constructing a data set comprising M×N groups of data, wherein each of the M×N groups of data corresponds to a photographing scene, wherein each photographing scene corresponds to a second light source and a to-be-photographed object, wherein M is a quantity of light sources, wherein N is a quantity of to-be-photographed objects, and wherein each of the M×N groups of data comprises third information in a corresponding photographing scene;
performing cluster analysis on spectra of M light sources in the data set to obtain a light source type corresponding to the second light source; and
further obtaining the training sample based on the third information and the light source type.

### Claim 4 (depends on 3)

The method of claim 3, wherein each of the M×N groups of data further comprises fourth information, wherein the fourth information comprises a third spectrum of a third light source in a corresponding photographing scene and an object material reflectivity spectrum corresponding to each pixel on a second multispectral sensor in the corresponding photographing scene, and wherein preforming the cluster analysis comprises:
obtaining the fourth information in each of the M×N groups of data to obtain M×N groups of fourth information; and
further performing the cluster analysis on the spectra based on the M×N groups of fourth information.

### Claim 5 (depends on 4)

The method of claim 4, wherein performing the cluster analysis on the spectra further comprises:
determining a classification standard of the M light sources based on the M×N groups of fourth information; and
further performing the cluster analysis on the spectra based on the classification standard.

### Claim 6 (depends on 5)

The method of claim 5, wherein determining the classification standard comprises:
selecting, for each to-be-photographed object in the N to-be-photographed objects, any two groups of fourth information from M groups of fourth information corresponding to the to-be-photographed object to obtain N*CM2 combinations;
calculating, for each combination, a two-dimensional data point corresponding to a related combination to obtain N*CM2 two-dimensional data points, wherein the two-dimensional data point comprises an average color distance and a light source spectral similarity indicator; and
further determining the classification standard based on the N*CM2 two-dimensional data points.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
calculating a first RGB value of each pixel based on one group of fourth information in the related combination;
calculating a second RGB value of each pixel based on a remaining group of fourth information in the related combination; and
determining, based on the first RGB value and the second RGB value, the average color distance corresponding to the related combination.

### Claim 8 (depends on 6)

The method of claim 6, further comprising determining, based on a fourth spectrum of a fourth light source comprised in one group of fourth information in the related combination and a fifth spectrum of a fifth light source comprised in a remaining group of fourth information in the related combination, the light source spectral similarity indicator corresponding to the related combination.

### Claim 9 (depends on 6)

The method of claim 6, further comprising:
drawing a two-dimensional intersection graph based on the N*CM2 two-dimensional data points;
determining a first parameter from a horizontal coordinate of the two-dimensional intersection graph to make a proportion of two-dimensional data points comprising vertical coordinates less than a preset value and horizontal coordinates less than the first parameter that exceed a preset threshold; and
setting the first parameter as the classification standard.

### Claim 10 (depends on 5)

The method of claim 5, wherein performing the cluster analysis on the spectra further comprises performing the cluster analysis on the spectra using a k-means clustering algorithm, wherein a light source spectral similarity indicator between any two cluster centers is greater than the classification standard, and wherein in each cluster, a spectral similarity indicator between any two light sources is less than the classification standard.

### Claim 11 (depends on 3)

The method of claim 3, wherein the first model comprises a branch configured to output the probability, and wherein the method further comprises setting the third information and the light source type as the training sample.

### Claim 12 (depends on 3)

The method of claim 3, wherein the first model comprises a first branch configured to output the probability and a second branch configured to output an object material reflectivity spectrum corresponding to each pixel in the current photographing scene, and wherein the method further comprises setting the third information, the light source type, and the object material reflectivity spectrum corresponding to each pixel on a second multispectral sensor in the corresponding photographing scene as the training sample.

### Claim 13 (depends on 12)

The method of claim 12, wherein the first branch comprises a first structure and a fully connected layer, wherein the method further comprises converting the first structure into the fully connected layer through downsampling for L times, wherein the first structure comprises a first convolutional layer, a first activation function, and a maximum pooling layer, wherein a first loss function of the first branch is cross entropy; wherein L is greater than or equal to 1, wherein the second branch comprises P residual blocks, wherein each of the P residual blocks comprises a second convolutional layer, a second activation function, a third convolutional layer, and a third activation function, wherein a second loss function of the second branch is L2, and wherein P is greater than or equal to 1.

### Claim 14 (depends on 1)

The method of claim 1, further comprising performing a summation on the second spectrum using the probability as a weighted sum weight to obtain the first spectrum.

### Claim 15

An electronic device comprising:
a red, green, and blue (RGB) sensor configured to generate a first image;
a multispectral sensor comprising a plurality of pixels configured to receive a light; and
one or more processors coupled to the RGB sensor and the multispectral sensor and configured to:
obtain first information in a current photographing scene, wherein the first information comprises at least one of the first image or a light intensity of the light;
input the first information into a first model to obtain a probability that a first light source in the current photographing scene belongs to each type of light source; and
determine a first spectrum of the first light source based on the probability and a second spectrum of each type of light source.

### Claim 16 (depends on 15)

The electronic device of claim 15, wherein before inputting the first information into the first model, the one or more processors are further configured to:
obtain a training sample comprising second information in different photographing scenes and light source types corresponding to light sources in the different photographing scenes; and
train the first model using the training sample.

### Claim 17 (depends on 16)

The electronic device of claim 16, wherein the one or more processors are further configured to:
construct a data set comprising M×N groups of data, wherein each of the M×N groups of data corresponds to a photographing scene, wherein each photographing scene corresponds to a second light source and a to-be-photographed object, wherein M is a quantity of light sources, wherein N is a quantity of to-be-photographed objects, and wherein each of the M×N groups of data comprises third information in a corresponding photographing scene;
perform cluster analysis on spectra of M light sources in the data set to obtain a light source type corresponding to the second light source; and
further obtain the training sample based on the third information and the light source type.

### Claim 18 (depends on 17)

The electronic device of claim 17, further comprising a second multispectral sensor comprising a plurality of second pixels, wherein each of the M×N groups of data further comprises fourth information, wherein the fourth information comprises a third spectrum of a third light source in a corresponding photographing scene and an object material reflectivity spectrum corresponding to each of the second pixels, and wherein the one or more processors are further configured to:
obtain the fourth information in each of the M×N group of data to obtain M×N groups of fourth information; and
further perform the cluster analysis on the spectra based on the M×N groups of fourth information.

### Claim 19 (depends on 18)

The electronic device of claim 18, wherein the one or more processors are further configured to:
determine a classification standard of the M light sources in the data set based on the M×N groups of forth information; and
further perform the cluster analysis on the spectra based on the classification standard.

### Claim 20 (depends on 19)

The electronic device of claim 19, wherein the one or more processors are further configured to:
select, for each to-be-photographed object in the N to-be-photographed objects, any two groups of fourth information from M groups of fourth information corresponding to the to-be-photographed object to obtain N*CM2 combinations;
calculate, for each combination, a two-dimensional data point corresponding to a related combination to obtain N*CM2 two-dimensional data points, wherein the two-dimensional data point comprises an average color distance and a light source spectral similarity indicator; and
further determine the classification standard based on the N*CM2 two-dimensional data points.

### Claim at position 21

-31. (canceled)