Method and system for reconstructing colour and depth information of a scene

A non-transitory, computer readable medium having computer executable instructions stored thereon, which when executed by a computer system, cause the computer system to perform a method of reconstructing colour and depth information of a scene by carrying out actions, comprising:

• receiving at least one colour image of a scene, the at least one colour image being captured from a respective viewpoint;
• obtaining depth information of the scene for the respective viewpoint;
• generating a point cloud of the scene based on the at least one colour image and the obtained depth information;
• generating a projection of the point cloud from an alternative viewpoint, the alternative viewpoint being different from the respective viewpoint, and including occluded parts of the scene for which colour and depth information is missing;
• converting the projection of the point cloud to respective sparse colour and depth images;
• estimating colour information for at least some of the scene in the sparse colour image for which there is no colour information, thereby reconstructing a more complete colour image of the scene from the alternative viewpoint;
• and
• estimating, based on the reconstructed colour image and the obtained depth information, depth information for at least some of the scene for which there is no depth information, thereby reconstructing a more complete depth image of the scene from the alternative viewpoint.

A system for reconstructing colour and depth information of a scene, the system comprising:

• a receiving unit configured to receive at least one colour image of a scene, the colour image being captured from a respective viewpoint;
• a depth unit operable to obtain a depth image for a corresponding portion of the scene;
• a point cloud generator operable to generate a point cloud from the colour image and obtained depth image;
• a projection unit configured to generate a projection of the point cloud from a viewpoint that is different from the respective viewpoint;
• a conversion unit operable to convert the generated projection into respective sparse colour and depth images, the sparse colour and depth images including occluded parts of the scene for which colour and depth information is missing;
• an in-painting processor configured to generate pixel values for at least some parts of the scene for which colour information is missing, thereby generating a reconstructed colour image;
• and
• a depth data generator configured to generate depth data for at least some parts of the scene for which depth data is missing, the depth data being generated based on the reconstructed colour image, thereby generating a reconstructed depth image.