Method and system for depth map reconstruction

A computing device, comprising:

• one or more non-transitory computer-readable storage media including instructions;
• and
• one or more processors coupled to the storage media, the one or more processors configured to execute the instructions to:
• access image data and depth data corresponding to one or more image frames to be displayed on an extended reality (XR) display device associated with the computing device;
• determine a plurality of sets of feature points corresponding to the one or more image frames based on a multi-layer sampling of the image data and the depth data;
• generate a set of sparse feature points based on an integration of the plurality of sets of feature points, wherein the set of sparse feature points are determined based on relative changes in depth data with respect to the plurality of sets of feature points, and wherein the set of sparse feature points are associated with a set of confidence scores, wherein each confidence score identifies how determinative a feature is for representing one or more objects in the one or more image frames;
• generate a set of sparse depth points based on the set of sparse feature points and the depth data;
• divide the set of sparse depth points into two or more levels of sparse depth points, wherein the sparse depth points are divided into each level based on the set of confidence scores associated with the sparse feature points;
• determine, based at least on a data throughput between the computing device and an XR device, which of one or more levels of sparse depth points to send to the XR device;
• and
• send the determined one or more levels of sparse depth points to the XR display device for reconstruction of a dense depth map corresponding to the one or more image frames utilizing the one or more levels of sparse depth points.

A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a computing device, cause the one or more processors to:

• access image data and depth data corresponding to one or more image frames to be displayed on an extended reality (XR) display device associated with the computing device;
• determine a plurality of sets of feature points corresponding to the one or more image frames based on a multi-layer sampling of the image data and the depth data;
• generate a set of sparse feature points based on an integration of the plurality of sets of feature points, wherein the set of sparse feature points are determined based on relative changes in depth data with respect to the plurality of sets of feature points, and wherein the set of sparse feature points are associated with a set of confidence scores wherein each confidence score identifies how determinative a feature is for representing one or more objects in the one or more image frames;
• generate a set of sparse depth points based on the set of sparse feature points and the depth data;
• divide the set of sparse depth points into two or more levels of sparse depth points, wherein the sparse depth points are divided into each level based on the set of confidence scores associated with the sparse feature points;
• determine, based at least on a data throughput between the computing device and an XR device, which of one or more levels of sparse depth points to send to the XR device;
• and
• send the determined one or more levels of sparse depth points to the XR display device for reconstruction of a dense depth map corresponding to the one or more image frames utilizing the one or more levels of sparse depth points.