# US 11,908,067 B1 · Method and device for gigapixel-level light field intelligent reconstruction of large-scale scene

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,067 B1
- Document ID: US11908067B1
- Status: Granted
- Kind: B1
- Application number: 18/351,742
- Filed: 2023-07-13
- Granted: 2024-02-20
- Earliest claimed: 2023-03-07
- Assignees: Tsinghua University
- Applicants: Tsinghua University
- Inventors: Lu Fang; Guangyu Wang; Jinzhi Zhang
- Web page: https://patentgrep.com/patent/US11908067B1

## CPC

- G06T 15/20: Perspective computation
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06V 10/40: Extraction of image or video features
- G06V 10/56: relating to colour
- G06V 10/755: {Deformable models or variational models, e.g. snakes or active contours}
- G06V 10/82: using neural networks
- G06V 20/64: Three-dimensional [3D] objects
- G06V 2201/10: Recognition assisted with metadata
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and a device for gigapixel-level light field intelligent reconstruction of a large-scale scene are provided. The method includes: obtaining a coarse three-dimensional geometric model based on a multi-view three-dimensional reconstruction system; constructing an implicit representation of the meta-deformed manifold on the coarse three-dimensional geometric model; and optimizing the implicit representation of the meta-deformed manifold to obtain the light field reconstruction in the form of free viewpoint rendering of the large-scale scene.

## Claims

### Claim 1

A method for gigapixel-level light field intelligent reconstruction of a large-scale scene, comprising:
obtaining a coarse three-dimensional geometric model based on a multi-view three-dimensional reconstruction system;
constructing an implicit representation of a meta-deformed manifold on the coarse three-dimensional geometric model, wherein the meta-deformed manifold is at least one of a manifold that is deformed through a meta deformation network or a manifold that is output from the meta deformation network; and
optimizing the implicit representation of the meta-deformed manifold to obtain a light field reconstruction in the form of a free viewpoint rendering of the large-scale scene;
wherein constructing the implicit representation of the meta-deformed manifold on the coarse three-dimensional geometric model comprises:
normalizing a two-dimensional parameter grid of local patches into a rectangle, inputting into the meta deformation network a feature vector obtained by interpolating a rectangular sampling point on the two-dimensional parameter grid, and outputting a deformation feature of the rectangular sampling point in a high-dimensional space;
inputting a high-dimensional feature vector obtained based on the deformation feature into a feature mapping network, and outputting a normal vector, a diffuse reflection color value, a non-Lambert material property and a geometric feature vector of the rectangular sampling point;
inputting a view direction vector of the rectangular sampling point, the normal vector of the rectangular sampling point, the non-Lambert material property and the geometric feature vector into a neural renderer, outputting a specular color value in a current viewing direction, and obtaining a color prediction result of the rectangular sampling point according to the specular color and the diffuse color;
wherein optimizing the implicit representation of the meta-deformed manifold to obtain the light field reconstruction in the form of free viewpoint rendering of the large-scale scene comprises:
constructing a multi-layer perceptron network model;
inputting coordinates of the rectangular sampling point, the feature vector obtained by interpolating and the view direction vector into the multi-layer perceptron network model, and outputting a color red green blue (RGB) value of the rectangular sampling point;
wherein the method further comprises:
acquiring sparse multi-view images of the large-scale scene;
randomly selecting a plurality of pixels on the sparse multi-view images to perform back projection, each pixel emitting a ray, calculating points on a coarse three-dimensional surface through which the ray of each pixel passes, and calculating a color value rendered at each point.

### Claim 2 (depends on 1)

The method according to claim 1, wherein obtaining the coarse three-dimensional geometric model based on the multi-view three-dimensional reconstruction system comprises:
acquiring a three-dimensional point cloud by using the multi-view three-dimensional reconstruction system;
obtaining a coarse three-dimensional surface model according to the three-dimensional point cloud and a Poisson surface reconstruction process; and
performing local conformal parameterization on the coarse three-dimensional surface model using a curved surface parameterization process to obtain a two-dimensional parameter grid of several local patches so as to obtain the coarse three-dimensional geometric model.