Computing high-resolution depth images using machine learning techniques

A non-transitory computer readable medium comprising computer code that, when executed by a processor of a computer, causes the processor to:

• receive a low resolution depth image captured from a scene, the low resolution depth image captured at an initial resolution;
• receive one or more color images captured from the scene;
• apply a machine learning model to the low resolution depth image and the one or more color images, where the machine learning model is trained to generate a high-resolution depth image from the low-resolution depth image and the one or more color images;
• and
• obtain from the model an upsampled depth image having an upsampled resolution higher than the initial resolution of the low resolution depth image.

A method comprising:

• generating a plurality of examples for a training data set, where generating each example in the training data set comprises:
• receiving a low-resolution depth image and one or more color images captured from a scene, where the low resolution depth image and the one or more color images were captured from a camera position in the scene,
• receiving three dimensional data captured that describes the scene,
• generating a three-dimensional reconstruction of the scene based on the three dimensional data, and
• generating a high-resolution depth image of the scene from the camera position based on the three-dimensional reconstruction of the scene, the high-resolution depth image having a higher resolution than the low-resolution depth image;
• and
• training a machine learning model using the training data set, where the inputs to the machine learning model comprise features extracted from the received low-resolution depth image and features extracted from the received one or more color images, and where the outputs of the machine learning model comprise the generated high-resolution depth image.