# US 2024/0187559 A1 · Apparatus and method with image processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0187559 A1
- Document ID: US20240187559A1
- Status: Published application
- Kind: A1
- Application number: 18/524,153
- Filed: 2023-11-30
- Published: 2024-06-06
- Earliest claimed: 2022-12-01
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Weiming LI; Inwoo HA; Qiang WANG; Young Hun SUNG; Hao WANG
- Web page: https://patentgrep.com/patent/US20240187559A1

## CPC

- G06T 15/00: Three-dimensional [3D] image rendering
- H04N 9/73: Colour balance circuits, e.g. white balance circuits or colour temperature control (camera processing pipelines for colour balance H04N23/88)
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/11: Region-based segmentation
- G06T 7/90: Determination of colour characteristics
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06T 15/506: {Illumination models}
- G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06T 2207/20208: High dynamic range [HDR] image processing
- G06T 2207/20221: Image fusion; Image merging
- G06V 20/05: Underwater scenes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A processor-implemented method includes obtaining an input image, predicting light source color information of a scene corresponding to the input image and a panoramic image corresponding to the input image using an image processing model, and generating a rendered image by rendering the input image based on either one or both of the light source color information and the panoramic image.

## Claims

### Claim 1

A processor-implemented method comprising:
obtaining an input image;
predicting light source color information of a scene corresponding to the input image and a panoramic image corresponding to the input image using an image processing model; and
generating a rendered image by rendering the input image based on either one or both of the light source color information and the panoramic image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the predicting of the light source color information of the scene corresponding to the input image and the panoramic image corresponding to the input image using the image processing model comprises:
generating an encoded feature of the input image by encoding the input image using an encoding network of the image processing model; and
predicting the light source color information and the panoramic image based on the encoded feature using a first prediction network of the image processing model.

### Claim 3 (depends on 1)

The method of claim 1, wherein the predicting of the light source color information of the scene corresponding to the input image and the panoramic image corresponding to the input image using the image processing model comprises:
generating an encoded feature of the input image by encoding the input image using an encoding network of the image processing model;
predicting the light source color information based on the encoded feature using a second prediction network of the image processing model; and
predicting the panoramic image by decoding the encoded feature using a decoding network of the image processing model.

### Claim 4 (depends on 2)

The method of claim 2, wherein the generating of the encoded feature of the input image by encoding the input image using the encoding network of the image processing model comprises:
obtaining a plurality of image blocks by dividing the input image;
generating encoded features of the plurality of image blocks by encoding each of the plurality of image blocks; and
generating an encoded feature of the input image based on the encoded features of the plurality of image blocks.

### Claim 5 (depends on 4)

The method of claim 4, wherein the generating of the encoded features of the plurality of image blocks by encoding each of the plurality of image blocks comprises:
generating an initial encoded feature of each image block by encoding each of the plurality of image blocks; and
improving a feature of the initial encoded feature of each image block using a first attention network of the image processing model to generate the improved encoded feature of each image block as encoded features of the plurality of image blocks.

### Claim 6 (depends on 4)

The method of claim 4, wherein the predicting of the light source color information comprises:
for each image block, predicting light source color information of a scene corresponding to the image block based on the encoded feature of the image block, and determining a reliability of the image block; and
obtaining the light source color information of the scene by pooling the light source color information corresponding to the image block based on the reliability of each image block.

### Claim 7 (depends on 4)

The method of claim 4, wherein the predicting of the panoramic image comprises:
obtaining a reliability of each image block;
obtaining a high dynamic range (HDR) image corresponding to each image block by decoding each of the encoded feature of each image block; and
generating the panoramic image by pooling the HDR image corresponding to the image block based on the reliability of each image block.

### Claim 8 (depends on 2)

The method of claim 2, wherein the generating of the encoded feature of the input image by encoding the input image using the encoding network of the image processing model comprises:
generating a global feature of the input image by encoding the input image using the encoding network;
generating a lighting feature and a scene feature by decoupling features from the global feature using a decoupling network of the image processing model; and
improving features of the lighting feature and the scene feature using a second attention network of the image processing model to generate a scene-based lighting feature as the encoded feature of the input image.

### Claim 9 (depends on 1)

The method of claim 1, wherein the generating of the rendered image by rendering the input image based on either one or both of the light source color information and the panoramic image comprises:
generating the rendered image by rendering the input image based on the input image, the panoramic image, and virtual object related information; and
outputting the rendered image using a video see through (VST) device.

### Claim 10 (depends on 1)

The method of claim 1, wherein the generating of the rendered image by rendering the input image based on either one or both of the light source color information and the panoramic image comprises:
generating a panoramic image corresponding to a white light source by performing color correction on the panoramic image based on the light source color information using a white balance method;
generating a first rendered image by rendering the input image based on the panoramic image of the white light source and virtual object related information;
generating actual light source color information of the scene by mapping the light source color information;
generating a second rendered image by adjusting a color of the first rendered image using the actual light source color information; and
outputting a target image formed by superposing the input image and the second rendered image using an optical see through (OST) device.

### Claim 11 (depends on 10)

The method of claim 10, wherein the target image is formed by superposing the input image and a prime number of the second rendered images.

### Claim 12 (depends on 1)

The method of claim 1, further comprising predicting camera parameters related to capturing of the input image using the image processing model.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
obtaining training data, wherein the training data comprises a limited view image with a different field of view under a same scene, a first panoramic image, and first light source color information corresponding to when the limited view image and the first panoramic image are captured;
predicting second light source color information of a scene corresponding to the limited view image and a second panoramic image corresponding to the limited view image using the image processing model; and
training the image processing model by adjusting network parameters of the image processing model based on the first light source color information, the second light source color information, the first panoramic image, and the second panoramic image.

### Claim 14

An electronic device comprising:
one or more processors configured to:
obtain an input image;
predict light source color information of a scene corresponding to the input image and a panoramic image corresponding to the input image using an image processing model; and
generate a rendered image by rendering the input image based on either one or both of the light source color information and the panoramic image.

### Claim 15 (depends on 14)

The electronic device of claim 14, wherein, for the predicting of the light source color information and the panoramic image, the one or more processors are configured to:
generate an encoded feature of the input image by encoding the input image using an encoding network of the image processing model; and
predict the light source color information and the panoramic image based on the encoded feature using a first prediction network of the image processing model.

### Claim 16 (depends on 14)

The electronic device of claim 14, wherein, for the predicting of the light source color information and the panoramic image, the one or more processors are configured to:
generate an encoded feature of the input image by encoding the input image using an encoding network of the image processing model;
predict the light source color information based on the encoded feature using a second prediction network of the image processing model; and
predict the panoramic image by decoding the encoded feature using a decoding network of the image processing model.

### Claim 17 (depends on 14)

The electronic device of claim 14, wherein, for the generating of the rendered image, the one or more processors are configured to:
generate the rendered image by rendering the input image based on the input image, the panoramic image, and virtual object related information; and
output the rendered image using a video see through (VST) device.

### Claim 18 (depends on 14)

The electronic device of claim 14, wherein, for the generating of the rendered image, the one or more processors are configured to:
generate a panoramic image corresponding to a white light source by performing color correction on the panoramic image based on the light source color information using a white balance method;
generate a first rendered image by rendering the input image based on the panoramic image of the white light source and virtual object related information;
generate actual light source color information of the scene by mapping the light source color information;
generate a second rendered image by adjusting a color of the first rendered image using the actual light source color information; and
output a target image formed by superposing the input image and the second rendered image using an optical see through (OST) device.

### Claim 19 (depends on 14)

The electronic device of claim 14, wherein the one or more processors are configured to:
obtain training data comprising a limited view image with a different field of view under a same scene, a first panoramic image, and first light source color information corresponding to when the limited view image and the first panoramic image are captured;
predict second light source color information of a scene corresponding to the limited view image and a second panoramic image corresponding to the limited view image using the image processing model; and
train the image processing model by adjusting network parameters of the image processing model based on the first light source color information, the second light source color information, the first panoramic image, and the second panoramic image.

### Claim 20

A processor-implemented method comprising:
obtaining training data, wherein the training data comprises a limited view image with a different field of view under a same scene, a first panoramic image, and first light source color information corresponding to when the limited view image and the first panoramic image are captured;
predicting second light source color information of a scene corresponding to the limited view image and a second panoramic image corresponding to the limited view image using an image processing model; and
training the image processing model by adjusting network parameters of the image processing model based on the first light source color information, the second light source color information, the first panoramic image, and the second panoramic image.
