Fusing multiple depth sensing modalities

A system comprising:

• a processor;
• and
• a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:
• receiving a first depth map comprising a first plurality of depth values and a second depth map comprising a second plurality of depth values;
• aligning at least part of the second depth map with a corresponding part of the first depth map;
• identifying a first aligned region of the first depth map having a different amount of depth information than a second aligned region of the second depth map;
• and
• inpainting the first aligned region of the first depth map based on depth values of the second aligned region of the second depth map by determining a plurality of scaling factors comprising, for each respective depth value of a plurality of pixels in the second aligned region, a corresponding scaling factor between the respective depth value and a corresponding depth value of the first aligned region, such that an output value of an optimization function is reduced below a threshold value, wherein the optimization function is configured to compare, for each respective depth value of the plurality of pixels in the second aligned region, (i) the respective depth value scaled according to the corresponding scaling factor and (ii) the corresponding depth value of the first aligned region.

A non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing system, cause the computing system to perform operations comprising:

• receiving a first depth map comprising a first plurality of depth values and a second depth map comprising a second plurality of depth values;
• aligning at least part of the second depth map with a corresponding part of the first depth map;
• identifying a first aligned region of the first depth map having a different amount of depth information than a second aligned region of the second depth map;
• and
• inpainting the first aligned region of the first depth map based on depth values of the second aligned region of the second depth map by determining a plurality of scaling factors comprising, for each respective depth value of a plurality of pixels in the second aligned region, a corresponding scaling factor between the respective depth value and a corresponding depth value of the first aligned region, such that an output value of an optimization function is reduced below a threshold value, wherein the optimization function is configured to compare, for each respective depth value of the plurality of pixels in the second aligned region, (i) the respective depth value scaled according to the corresponding scaling factor and (ii) the corresponding depth value of the first aligned region.