US 11,238,604 B1 · Densifying sparse depth maps

Abstract

A system and techniques that use one or more machine learning models to predict a dense depth map (e.g., of depth values for all pixels or at least more pixels than a sparse estimation source (e.g., SLAM)). In some implementations, the machine learning model includes two sub models (e.g., neural networks). The first machine learning model predicts computer vision data such as semantic labels and surface normal directions from an input image. This computer vision data will be used to add to or otherwise improve sparse depth data. Specifically, a second machine learning model takes the semantic labels and surface normal directions from and sparse depth data (e.g., 3D points) from a sparse point estimation source (e.g., SLAM) as inputs and outputs a depth map. The output depth map effectively densities the initial depth data (e.g., from SLAM) by providing depth data for additional pixels of the image.

Claims20

Dependent claims

A method, comprising:

at an electronic device having a processor:
obtaining an image of a physical setting from an image capture device;
producing a semantic prediction and a surface normal prediction using a first machine learning model with input comprising the image;
producing depth data using a depth estimator on the image;
and
outputting a depth map using a second machine learning model with input comprising the semantic prediction, the surface normal prediction, and the depth data.

3 more elements

Dependencies are inferred from the claim text.

Densifying sparse depth maps

Life of this patent

Densifying sparse depth maps

Life of this patent

Abstract

Claims20