Back to results
US 2021/0279904 A1Published application
Systems and methods for depth estimation by learning triangulation and densification of sparse points for multi-view stereo
AssigneeMagic Leap, Inc.
InventorAyan Tuhinendu SINHA
Life of this patent
Estimating the term…
Abstract
Systems and methods for estimating depths of features in a scene or environment surrounding a user of a spatial computing system, such as a virtual reality, augmented reality or mixed reality (collectively, cross reality) system, in an end-to-end process. The estimated depths can be utilized by a spatial computing system, for example, to provide an accurate and effective 3D cross reality experience.
Claims16
A method for estimating depth of features in a scene from multi-view images, the method comprising:
- obtaining multi-view images, including an anchor image of the scene and a set of reference images of the scene;
- passing the anchor image and reference images through a shared RGB encoder and descriptor decoder which (1) outputs a respective descriptor field of descriptors for the anchor image and each reference image, (ii) detects interest points in the anchor image in conjunction with relative poses to determine a search space in the reference images from alternate view-points, and (iii) outputs intermediate feature maps;
- sampling the respective descriptors in the search space of each reference image to determine descriptors in the search space and matching the identified descriptors with descriptors for the interest points in the anchor image, such matched descriptors referred to as matched keypoints;
- triangulating the matched keypoints using singular value decomposition (SVD) to output 3D points;
- passing the 3D points through a sparse depth encoder to create a sparse depth image from the 3D points and output feature maps;
- and
- a depth decoder generating a dense depth image based on the output feature maps for the sparse depth encoder and the intermediate feature maps from the RGB encoder.
4 more elements
A cross reality system, comprising:
- a head-mounted display device having a display system;
- a computing system in operable communication with the head-mounted display;
- a plurality of camera sensors in operable communication with the computing system;
- wherein the computing system is configured to estimate depths of features in a scene from a plurality of multi-view images captured by the camera sensors by a process comprising:
- obtaining a multi-view images, including an anchor image of the scene and a set of reference images of a scene within a field of view of the camera sensors from the camera sensors;
- passing the anchor image and reference images through a shared RGB encoder and descriptor decoder which (1) outputs a respective descriptor field of descriptors for the anchor image and each reference image, (ii) detects interest points in the anchor image in conjunction with relative poses to determine a search space in the reference images from alternate view-points, and (iii) outputs intermediate feature maps;
- sampling the respective descriptors in the search space of each reference image to determine descriptors in the search space and matching the identified descriptors with descriptors for the interest points in the anchor image, such matched descriptors referred to as matched keypoints;
- triangulating the matched keypoints using singular value decomposition (SVD) to output 3D points;
- passing the 3D points through a sparse depth encoder to create a sparse depth image from the 3D points and output feature maps;
- and
- a depth decoder generating a dense depth image based on the output feature maps for the sparse depth encoder and the intermediate feature maps from the RGB encoder.
8 more elements
Dependencies are inferred from the claim text.