# US 2021/0241495 A1 · Method and system for reconstructing colour and depth information of a scene

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241495 A1
- Document ID: US20210241495A1
- Status: Published application
- Kind: A1
- Application number: 17/268,856
- Filed: 2019-08-06
- Published: 2021-08-05
- Earliest claimed: 2018-08-23
- Assignees: Sony Interactive Entertainment Inc.
- Applicants: Sony Interactive Entertainment Inc.
- Inventors: Fabio Cappello; Nigel John Williams
- Web page: https://patentgrep.com/patent/US20210241495A1

## CPC

- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 7/90: Determination of colour characteristics
- G06T 7/50: Depth or shape recovery
- G06T 15/205: {Image-based rendering}
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30168: Image quality inspection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method of reconstructing colour and depth information of a scene includes receiving a colour image of a scene and obtaining depth information of the scene. The colour and depth images are used to generate a point cloud, which is then projected to an alternative viewpoint and converted to sparse colour and depth images. Colour information is then estimated for at least some parts of the sparse colour image, resulting in a reconstructed colour image. The reconstructed colour image is used with the existing depth information to estimate depth information for the sparse depth image. In this way, colour and depth information of the scene can be estimated, and used to generate colour and depth images of the scene from a desired viewpoint. A corresponding system for reconstructing colour and depth information is also provided.

## Claims

### Claim 1

A method of reconstructing colour and depth information of a scene, the method comprising:
receiving at least one colour image of a scene, the at least one colour image being captured from a respective viewpoint;
obtaining depth information of the scene for the respective viewpoint;
generating a point cloud of the scene based on the at least one colour image and the obtained depth information;
generating a projection of the point cloud from an alternative viewpoint, the alternative viewpoint being different from the respective viewpoint, and including occluded parts of the scene for which colour and depth information is missing;
converting the projection of the point cloud to respective sparse colour and depth images;
estimating colour information for at least some of the scene in the sparse colour image for which there is no colour information, thereby reconstructing a more complete colour image of the scene from the alternative viewpoint; and
estimating, based on the reconstructed colour image and the obtained depth information, depth information for at least some of the scene for which there is no depth information, thereby reconstructing a more complete depth image of the scene from the alternative viewpoint.

### Claim 2 (depends on 1)

A method according to claim 1, wherein estimating the colour information comprises inputting the sparse colour image to a trained machine learning model, the machine learning model being trained with complete and incomplete colour image pairs, and trained to estimate pixel values for incomplete parts of an image.

### Claim 3 (depends on 2)

A method according to claim 2, wherein the machine learning model comprises a fully convolutional network model.

### Claim 4 (depends on 1)

A method according to claim 1, wherein estimating the depth information comprises:
estimating surface normals of one or more surfaces in the reconstructed colour image;
detecting one or more occlusion boundaries in the reconstructed colour image; and
estimating the depth information based on the estimated surface normals, the detected occlusion boundaries and the obtained depth information.

### Claim 5 (depends on 4)

A method according to claim 4, comprising inputting the reconstructed colour image to a neural network model, the neural network model being trained to estimate surface normals and occlusion boundaries in colour images; and
wherein estimating the depth information comprises using the estimated surface normals, detected occlusion boundaries and the obtained depth information to guide a global optimization process.

### Claim 6 (depends on 1)

A method according to claim 1, wherein generating the point cloud comprises applying an inverse projection to the at least one colour image and obtained depth information.

### Claim 7 (depends on 1)

A method according to claim 1, comprising outputting the more complete colour image for display at a display device.

### Claim 8 (depends on 1)

A method according to claim 1, comprising:
receiving a plurality of colour images, each colour image being captured from a different respective viewpoint;
obtaining depth information for each of the respective viewpoints;
generating the point cloud based on the plurality of captured colour images and the obtained depth information;
generating a projection of the point cloud from an alternative viewpoint, the alternative viewpoint being different from each of the respective viewpoints and including occluded parts of the scene for which colour and depth information is missing;
converting the projection of the point cloud to respective sparse colour and depth images;
estimating colour information for at least some of the scene in the sparse colour image for which there is no colour information, thereby reconstructing a more complete colour image of the scene from the alternative viewpoint; and
estimating, based on the reconstructed colour image and the obtained depth information, depth information for at least some of the scene for which there is no depth information, thereby reconstructing a more complete depth image of the scene from the alternative viewpoint.

### Claim 9 (depends on 1)

A method according to claim 1, comprising successive reconstructions of colour images and depth images;
wherein the step of receiving at least one colour image of a scene comprises receiving a previously reconstructed colour image, and the step of obtaining depth information of a scene comprises receiving previously reconstructed depth image.

### Claim 10 (depends on 9)

A method according to claim 9, comprising:
determining an error associated with the reconstructed colour and depth images; and
wherein successive reconstructions of colour and depth images end when either (i) the error associated with a current reconstructed colour image and depth image is below a threshold value, or (ii) a threshold number of successive reconstructions have occurred and the error associated with the current reconstructed colour image and depth image exceeds the threshold value.

### Claim 11

A non-transitory, computer readable medium having computer executable instructions stored thereon, which when executed by a computer system, cause the computer system to perform a method of reconstructing colour and depth information of a scene by carrying out actions, comprising:
receiving at least one colour image of a scene, the at least one colour image being captured from a respective viewpoint;
obtaining depth information of the scene for the respective viewpoint;
generating a point cloud of the scene based on the at least one colour image and the obtained depth information;
generating a projection of the point cloud from an alternative viewpoint, the alternative viewpoint being different from the respective viewpoint, and including occluded parts of the scene for which colour and depth information is missing;
converting the projection of the point cloud to respective sparse colour and depth images;
estimating colour information for at least some of the scene in the sparse colour image for which there is no colour information, thereby reconstructing a more complete colour image of the scene from the alternative viewpoint; and
estimating, based on the reconstructed colour image and the obtained depth information, depth information for at least some of the scene for which there is no depth information, thereby reconstructing a more complete depth image of the scene from the alternative viewpoint.

### Claim 12

A system for reconstructing colour and depth information of a scene, the system comprising:
a receiving unit configured to receive at least one colour image of a scene, the colour image being captured from a respective viewpoint;
a depth unit operable to obtain a depth image for a corresponding portion of the scene;
a point cloud generator operable to generate a point cloud from the colour image and obtained depth image;
a projection unit configured to generate a projection of the point cloud from a viewpoint that is different from the respective viewpoint;
a conversion unit operable to convert the generated projection into respective sparse colour and depth images, the sparse colour and depth images including occluded parts of the scene for which colour and depth information is missing;
an in-painting processor configured to generate pixel values for at least some parts of the scene for which colour information is missing, thereby generating a reconstructed colour image; and
a depth data generator configured to generate depth data for at least some parts of the scene for which depth data is missing, the depth data being generated based on the reconstructed colour image, thereby generating a reconstructed depth image.

### Claim 13 (depends on 12)

A system according to claim 12, wherein the point cloud generator is configured to receive the reconstructed colour image and the reconstructed depth image; and
wherein the projection unit is configured to generate a projection of the point cloud from a further different viewpoint, based on the received colour and depth images, and the reconstructed colour and depth images.

### Claim 14 (depends on 12)

A system according to claim 12, wherein the in-painting processor comprises a machine learning model, the machine learning model being trained to generate pixel values for incomplete images; and
wherein the machine learning model is configured to receive the sparse colour image, and in response thereto, generate pixel values for at least some parts of the scene for which colour information is missing.

### Claim 15 (depends on 12)

A system according to claim 12, wherein the depth data generator comprises:
a surface normal estimator operable to estimate a surface normal of at least one surface in the reconstructed colour image;
an occlusion boundary detector configured to detect an occlusion boundary in the reconstructed colour image; and
wherein the depth data generator is configured to generate depth data based on the at least one estimated surface normal, detected occlusion boundary and existing depth information for the scene.

### Claim 16 (depends on 15)

A system according to claim 15, wherein the surface normal estimator and occlusion boundary detector comprise a neural network, the neural networking being trained to estimate surface normals and occlusion boundaries in colour images.
