Feed-forward structure-from-motion via latent global alignment
OwnerSven Harald Adolf Elflein, Sergio Da Silva Agostinho +2inventors
InventorsSven Harald Adolf Elflein, Sergio Da Silva Agostinho, Qunjie Zhou, Laura Leal-Taixe
Life of this patent
Estimating the term…
Abstract
Systems and methods for performing three-dimensional (3D) scene reconstruction based on a set of images. According to one or more embodiments, neural network architectures and machine learning techniques are provided for performing global alignment in latent space to share context information across the input images and reconstructing selective image pairs according to relevant correspondences between images, thereby enabling robust, accurate, and efficient global alignment for a variety of computer vision applications, e.g., 3D reconstruction.
Claims20
A system, comprising:
- one or more memories to store parameters corresponding to one or more neural networks;
- and
- one or more processors to use one or more neural networks configured with the parameters to generate a set of image tokens based on a plurality of input images, process the set of image tokens using an alignment network to generate a set of aligned image tokens, generate a scene graph that represents image connections between two or more of the plurality of input images, decode subsets of the aligned image tokens based on the image connections to produce a plurality of pairwise point maps, and generate a plurality of 3D pointmaps representing the plurality of input images.
A method for performing three-dimensional (3D) scene reconstruction comprising:
- generating, based on a plurality of input images, a set of image tokens;
- processing, via an alignment network, the set of image tokens to generate a set of aligned image tokens;
- generating a scene graph representing one or more image connections between the plurality of input images;
- decoding, based on the one or more image connections, subsets of the aligned image tokens to produce a plurality of pairwise 3D pointmaps corresponding to input image pairs;
- and
- generating, based on the plurality of pairwise pointmaps and the scene graph, a plurality of 3D pointmaps corresponding to the plurality of input images.
One or more processors comprising:
- circuitry to implement an application programming interface (API), which in response to an API call received through the API, one or more operations comprising:
- an operation to receive a plurality of input images;
- an operation to generate a set of image tokens based on the plurality of input images;
- an operation to process, via an alignment network, the set of image tokens to generate a set of aligned image tokens;
- an operation to generate a scene graph representing one or more image connections between the plurality of input images;
- an operation to decode, based on the one or more image connections, subsets of the aligned image tokens to produce a plurality of pairwise 3D pointmaps corresponding to input image pairs;
- and
- an operation to generate, based on the plurality of pairwise pointmaps and the scene graph, a plurality of 3D pointmaps corresponding to the plurality of input images.
Dependencies are inferred from the claim text.