# US 2023/0267678 A1 · Device and method for image processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267678 A1
- Document ID: US20230267678A1
- Status: Published application
- Kind: A1
- Application number: 18/311,053
- Filed: 2023-05-02
- Published: 2023-08-24
- Earliest claimed: 2020-11-06
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Pedro Vieira de CASTRO; Manolis VASILEIADIS; Ales LEONARDIS; Benjamin BUSAM
- Web page: https://patentgrep.com/patent/US20230267678A1

## CPC

- G06T 15/506: {Illumination models}
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 7/50: Depth or shape recovery
- G06T 15/04: Texture mapping
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2215/12: Shadow map, environment map
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A device comprising an image processor apparatus, the image processor apparatus being configured for implementing an image based computational model as part of an end-to-end processing pipeline. The device is configured to operate by receiving colour-specific image data representing a scene, receiving depth data of the scene, processing the colour-specific image data using the image based computational model to form a feature map of the scene, and forming in dependence on the feature map and the depth data an illumination map representing an estimation of the illumination on a set of three-dimensional locations in the scene.

## Claims

### Claim 1

A device comprising an image processor apparatus, the image processor apparatus being configured for implementing an image based computational model as part of an end-to-end processing pipeline, the device is configured to perform:
receiving colour-specific image data representing a scene;
receiving depth data of the scene;
processing the colour-specific image data using the image based computational model to form a feature map of the scene; and
forming in dependence on the feature map and the depth data an illumination map representing an estimate of the illumination on a set of three-dimensional locations in the scene.

### Claim 2 (depends on 1)

The device according to claim 1, wherein the image based computational model is a neural network model.

### Claim 3 (depends on 1)

The device according to claim 1, wherein the colour-specific image data is received from a camera of the device.

### Claim 4 (depends on 1)

The device according to claim 1, wherein the depth data is received from a depth sensor of the device.

### Claim 5 (depends on 1)

The device according to claim 1, wherein the depth data is received as an estimate based on the colour-specific image data.

### Claim 6 (depends on 1)

The device according to claim 1, wherein determining an illumination at a selected location within the scene comprises shifting a frame of reference of the illumination map to be centred on the selected location and combining the illumination points of the illumination map based on their spatial distribution around the selected location at the centre of the frame of reference.

### Claim 7 (depends on 6)

The device according to claim 6, wherein multiple selected locations can be represented simultaneously by implementing a shifting of the reference frame.

### Claim 8 (depends on 1)

The device according to claim 1, wherein the illumination map comprises a plurality of illumination points, each illumination point representing for a corresponding pixel of the colour-specific image one of (i) an illumination level or (ii) an illumination hue.

### Claim 9 (depends on 8)

The device according to claim 8, wherein the illumination map further comprises data representing a depth corresponding to each illumination point.

### Claim 10 (depends on 1)

The device according to claim 1, wherein a feature vector representation of the illumination at the selected location is extracted from the illumination map by an extraction neural network model.

### Claim 11

A computer-implemented method for processing an image by means of an image processor apparatus configured for implementing an image based computational model as part of an end-to-end processing pipeline, the method comprising:
receiving colour-specific image data representing a scene;
receiving depth data of the scene;
processing the colour-specific image data using the image based computational model to form a feature map of the scene; and
forming in dependence on the feature map and the depth data an illumination map representing an estimate of the illumination on a set of three-dimensional locations in the scene.

### Claim 12 (depends on 11)

The method according to claim 11, comprising determining an illumination at a selected location within the scene by:
shifting a frame of reference of the illumination map to be centred on the selected location; and
combining the illumination points of the illumination map based on their spatial distribution around the selected location at the centre of the frame of reference.

### Claim 13 (depends on 12)

The method according to claim 12, comprising extracting a feature vector representation of the illumination at the selected location from the illumination map by an extraction neural network model.

### Claim 14 (depends on 13)

The method according to claim 13, comprising processing the feature vector representation to generate a colour-specific spherical harmonic representation, a depth spherical harmonic representation, and an indication of a geometry distance estimate of the illumination at the selected location.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the indication of the geometry distance estimate comprises one or more spherical harmonic coefficients.

### Claim 16 (depends on 14)

The method according to claim 14, wherein the spherical harmonic representations each comprise 36 coefficients representing a respective degree of approximation each multiplied by 3 colour channels.

### Claim 17 (depends on 11)

The method according to claim 11, comprising implementing a discriminator neural network to validate the output of the processing pipeline by:
distinguishing the feature vectors corresponding to synthetic images from the feature vectors corresponding to real images to produce a gradient;
processing the gradient by a gradient reversal layer; and
using the processed gradient to optimize the image based computational model and extraction neural network model.

### Claim 18 (depends on 11)

The method according to claim 11, wherein the image based computational model is a neural network.
