Method and device for gigapixel-level light field intelligent reconstruction of large-scale scene

A method for gigapixel-level light field intelligent reconstruction of a large-scale scene, comprising:

• obtaining a coarse three-dimensional geometric model based on a multi-view three-dimensional reconstruction system;
• constructing an implicit representation of a meta-deformed manifold on the coarse three-dimensional geometric model, wherein the meta-deformed manifold is at least one of a manifold that is deformed through a meta deformation network or a manifold that is output from the meta deformation network;
• and
• optimizing the implicit representation of the meta-deformed manifold to obtain a light field reconstruction in the form of a free viewpoint rendering of the large-scale scene;
• wherein constructing the implicit representation of the meta-deformed manifold on the coarse three-dimensional geometric model comprises:
• normalizing a two-dimensional parameter grid of local patches into a rectangle, inputting into the meta deformation network a feature vector obtained by interpolating a rectangular sampling point on the two-dimensional parameter grid, and outputting a deformation feature of the rectangular sampling point in a high-dimensional space;
• inputting a high-dimensional feature vector obtained based on the deformation feature into a feature mapping network, and outputting a normal vector, a diffuse reflection color value, a non-Lambert material property and a geometric feature vector of the rectangular sampling point;
• inputting a view direction vector of the rectangular sampling point, the normal vector of the rectangular sampling point, the non-Lambert material property and the geometric feature vector into a neural renderer, outputting a specular color value in a current viewing direction, and obtaining a color prediction result of the rectangular sampling point according to the specular color and the diffuse color;
• wherein optimizing the implicit representation of the meta-deformed manifold to obtain the light field reconstruction in the form of free viewpoint rendering of the large-scale scene comprises:
• constructing a multi-layer perceptron network model;
• inputting coordinates of the rectangular sampling point, the feature vector obtained by interpolating and the view direction vector into the multi-layer perceptron network model, and outputting a color red green blue (RGB) value of the rectangular sampling point;
• wherein the method further comprises:
• acquiring sparse multi-view images of the large-scale scene;
• randomly selecting a plurality of pixels on the sparse multi-view images to perform back projection, each pixel emitting a ray, calculating points on a coarse three-dimensional surface through which the ray of each pixel passes, and calculating a color value rendered at each point.