Fast large-scale radiance field reconstruction
AssigneesAdobe Inc., The Regents of the University of Calfornia
InventorsZexiang Xu, Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su
Life of this patent
Estimating the term…
Abstract
Embodiments are disclosed for fast large-scale radiance field reconstruction. A method of fast large-scale radiance field reconstruction may include receiving a sequence of input images that depict views of a scene and extracting, using an image encoder, image features from the sequence of input images. A first one or more machine learning models may generate a local volume based on the image features corresponding to one or more images from the sequence of input images. A second one or more machine learning models may generate a global volume based on the local volume. A novel view of the scene is synthesized based on the global volume.
Claims20
A method comprising:
- receiving a sequence of input images that depict views of a scene;
- extracting, using an image encoder, image features for an image from the sequence of input images;
- generating, using a first one or more machine learning models, a local volume for the image from the sequence of input images based on the image features corresponding to one or more images from the sequence of input images, wherein the local volume defines a local radiance field;
- generating, using a second one or more machine learning models, an iterative volume based on the local volume for the image from the sequence of input images and a previous image from the sequence of input images, wherein a combination of the iterative volume is a global volume, wherein the global volume has a global radiance field, and wherein the first one or more machine learning models and second one or more machine learning models are trained end to end using a loss based on the local radiance field and the global radiance field;
- and
- synthesizing a novel view of the scene based on the global volume.
A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
- receiving a sequence of input images that depict views of a scene;
- extracting, using an image encoder, image features for an image from the sequence of input images;
- generating, using a first one or more machine learning models, a local volume for the image from the sequence of input images based on the image features corresponding to one or more images from the sequence of input images, wherein the local volume defines a local radiance field;
- generating, using a second one or more machine learning models, an iterative volume based on the local volume for the image from the sequence of input images and a previous image from the sequence of input images, wherein a combination of the iterative volume is a global volume, wherein the global volume has a global radiance field, and wherein the first one or more machine learning models and second one or more machine learning models are trained end to end using a loss based on the local radiance field and the global radiance field;
- and
- synthesizing a novel view of the scene based on the global volume.
A system comprising:
- a memory component;
- and
- a processing device coupled to the memory component, the processing device to perform operations comprising:
- receiving a sequence of input images that depict views of a scene;
- extracting, using an image encoder, image features for an image from the sequence of input images;
- generating, using a first one or more machine learning models, a local volume for the image from the sequence of input images based on the image features corresponding to one or more images from the sequence of input images, wherein the local volume defines a local radiance field;
- generating, using a second one or more machine learning models, an iterative volume based on the local volume for the image from the sequence of input images and a previous image from the sequence of input images, wherein a combination of the iterative volume is a global volume, wherein the global volume has a global radiance field, and wherein the first one or more machine learning models and second one or more machine learning models are trained end to end using a loss based on the local radiance field and the global radiance field;
- and
- synthesizing a novel view of the scene based on the global volume.
Dependencies are inferred from the claim text.