# US 2015/0146971 A1 · Mesh reconstruction from heterogeneous sources of data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0146971 A1
- Document ID: US20150146971A1
- Status: Published application
- Kind: A1
- Application number: 14/555,409
- Filed: 2014-11-26
- Published: 2015-05-28
- Earliest claimed: 2013-11-27
- Assignees: Autodesk, Inc.
- Applicants: Autodesk, Inc.
- Inventors: Luc Franck Robert; Emmanuel Gallo
- Web page: https://patentgrep.com/patent/US20150146971A1

## CPC

- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 7/557: from light fields, e.g. from plenoptic cameras
- G06T 7/593: from stereo images
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/10008: from scanner, fax or copier
- G06T 2207/10012: Stereo images
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A system, apparatus, method, computer program product, and computer readable storage medium provide the ability to reconstruct a surface mesh. Photo image data is obtained from a set of overlapping photographic images. Scan data is obtained from a scanner. A point cloud is generated from a combination of the photo image data and the scan data. An initial rough mesh is estimated from the point cloud data. The initial rough mesh is iteratively refined into a refined mesh.

## Claims

### Claim 1

A computer-implemented method for reconstructing a surface mesh, comprising:
obtaining photo image data from a set of overlapping photographic images;
obtaining scan data from a scanner;
generating a point cloud from a combination of the overlapping photo image data and the scan data;
estimating an initial rough mesh from the point cloud data; and
refining the initial rough mesh into a refined mesh.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the generating the point cloud from the overlapping photo image data comprises:
evaluating a photo-consistency score between the set of overlapping photographic images;
projecting the set of overlapping photographic images onto a moving fronto-parallel plane; and
maximizing matching photo-consistency scores locally over a depth of the set of overlapping photographic images.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the generating the point cloud from the scan data comprises:
collecting three-dimensional (3D) points directly from a scan by the scanner;
simplifying the scan by removing redundant data.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the generating the point cloud comprises:
refining the point cloud;
clustering the point cloud; and
filtering the point cloud to remove outliers.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the refining the initial rough mesh into the refined mesh utilizes an error function that is based on a signed surface distance to the point cloud for the scan data.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the refining the initial rough mesh into the refined mesh comprises a combination of:
the error function; and
a surface reprojection error for the photo image data, wherein the surface reprojection error is based on a photo-consistency score between the set of overlapping photographic images back-projected onto the initial rough mesh.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating a textured surface from the refined mesh based on red, green, blue (RGB) data from the photo image data and/or the scan data; and
displaying the textured surface.

### Claim 8

A non-transitory computer readable storage medium encoded with computer program instructions which when accessed by a computer cause the computer to load the program instructions to a memory therein creating a special purpose data structure causing the computer to operate as a specially programmed computer, executing a method of reconstructing a surface mesh, comprising:
obtaining, in the specially programmed computer, photo image data from a set of overlapping photographic images;
obtaining, in the specially programmed computer, scan data from a scanner;
generating, in the specially programmed computer, a point cloud from a combination of the overlapping photo image data and the scan data;
estimating, in the specially programmed computer, an initial rough mesh from the point cloud data; and
refining, in the specially programmed computer, the initial rough mesh into a refined mesh.

### Claim 9 (depends on 8)

The non-transitory computer readable storage medium of claim 8, wherein the generating the point cloud from the overlapping photo image data comprises:
evaluating, in the specially programmed computer, a photo-consistency score between the set of overlapping photographic images;
projecting, in the specially programmed computer, the set of overlapping photographic images onto a moving fronto-parallel plane; and
maximizing, in the specially programmed computer, matching photo-consistency scores locally over a depth of the set of overlapping photographic images.

### Claim 10 (depends on 8)

The non-transitory computer readable storage medium of claim 8, wherein the generating the point cloud from the scan data comprises:
collecting, in the specially programmed computer, three-dimensional (3D) points directly from a scan by the scanner;
simplifying, in the specially programmed computer, the scan by removing redundant data.

### Claim 11 (depends on 8)

The non-transitory computer readable storage medium of claim 8, wherein the generating the point cloud comprises:
refining, in the specially programmed computer, the point cloud;
clustering, in the specially programmed computer, the point cloud; and
filtering, in the specially programmed computer, the point cloud to remove outliers.

### Claim 12 (depends on 8)

The non-transitory computer readable storage medium of claim 8, wherein the refining the initial rough mesh into the refined mesh utilizes an error function that is based on a signed surface distance to the point cloud for the scan data.

### Claim 13 (depends on 12)

The non-transitory computer readable storage medium of claim 12, wherein the refining the initial rough mesh into the refined mesh comprises a combination of:
the error function; and
a surface reprojection error for the photo image data, wherein the surface reprojection error is based on a photo-consistency score between the set of overlapping photographic images back-projected onto the initial rough mesh.

### Claim 14 (depends on 8)

The non-transitory computer readable storage medium of claim 8, further comprising:
generating, in the specially programmed computer, a textured surface from the refined mesh based on red, green, blue (RGB) data from the photo image data and/or the scan data; and
displaying, via the specially programmed computer, the textured surface.
