# US 10,319,080 B2 · Point cloud noise and outlier removal for image-based 3D reconstruction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,319,080 B2 - Document ID: US10319080B2 - Status: Granted - Kind: B2 - Application number: 16/029,328 - Filed: 2018-07-06 - Published: 2018-11-01 - Granted: 2019-06-11 - Earliest claimed: 2016-09-30 - Assignees: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich) - Applicants: Disney Enterprises, Inc. - Inventors: Changil Kim; Olga Sorkine-Hornung; Christopher Schroers; Henning Zimmer; Katja Wolff; Mario Botsch; Alexander Sorkine-Hornung - Web page: https://patentgrep.com/patent/US10319080B2 ## CPC - G06T 5/70: Denoising; Smoothing - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 15/06: Ray-tracing - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2210/56: Particle system, point based geometry or rendering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Enhanced removing of noise and outliers from one or more point sets generated by image-based 3D reconstruction techniques is provided. In accordance with the disclosure, input images and corresponding depth maps can be used to remove pixels that are geometrically and/or photometrically inconsistent with the colored surface implied by the input images. This allows standard surface reconstruction methods (such as Poisson surface reconstruction) to perform less smoothing and thus achieve higher quality surfaces with more features. In some implementations, the enhanced point-cloud noise removal in accordance with the disclosure can include computing per-view depth maps, and detecting and removing noisy points and outliers from each per-view point cloud by checking if points are consistent with the surface implied by the other input views. ## Claims ### Claim 1 A method for reconstructing a three-dimensional (3D) model of an object, the method being performed by a computer system and comprising: receiving images of the object, the images capturing the object from different viewpoints; obtaining a depth map for each of the images, the depth map including information relating to a surface of the object in a 3D space; projecting each pixel point in each depth map into the 3D space to obtain a point set; for each point in the point set, determining whether the point is consistent with the surface of the object in the 3D space as indicated by a set of depth maps for the images, wherein the consistency determination includes: for each depth map in the set: performing an intersection of a ray with the surface of the object indicated by the depth map to find an intersection point; and in response to the intersection point being found, determining a distance between the point and intersection point as a distance between the point and the surface of the object as seen from a camera viewpoint corresponding to the depth map; determining whether the point is consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object; and determining whether to discard the point from point set based on the determination whether the point is consistent with the surface of the object; and reconstructing the 3D model using the point set. ### Claim 2 (depends on 1) The method of claim 1, wherein the determination whether the point is consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes: averaging distances between the point and the surface of the object as indicated by the set of depth maps to obtain an average distance between the point and the surface of the object. ### Claim 3 (depends on 2) The method of claim 2, wherein the determination whether the point is consistent with the surface of the object is further based on a threshold distance value such that the point is determined to be consistent when the average distance between the point and the surface of the object is less than or equal to the threshold distance value. ### Claim 4 (depends on 1) The method of claim 1, wherein whether the point is consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes: truncating any distance between the point and the surface as indicated by a given depth map in the set to a preset value when the distance exceeds a threshold value. ### Claim 5 (depends on 1) The method of claim 1, wherein determining whether the point is consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes: assigning a weight to each of the determined distances, and determining the distance between the point and the surface of the object based on the weighted distances. ### Claim 6 (depends on 1) The method of claim 1, wherein determining whether the point is consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes: determining whether any distance between the point and the surface as indicated by a given depth map in the set is smaller than a threshold value indicating a negative distance value, and in response to determining the distance is smaller than the threshold value, excluding such a distance from the determination whether the point is consistent with the surface of the object. ### Claim 7 (depends on 1) The method of claim 1, further comprising: for each point in the point set, determining whether the point is photometrically consistent with the surface of the object based on the intersection points found; and, wherein the determination whether to discard the point from the point set is further based on whether the point is photometrically consistent the surface of the object. ### Claim 8 (depends on 7) The method of claim 7, wherein determining whether the point is photometrically consistent with the surface of the object based on the intersection points found includes: for each intersection point found: determining whether the distance between the point and intersection point is larger than a threshold value; and in response to determining the distance between the point and intersection point is larger than the threshold value, excluding the intersection point from the determination whether the point is photometrically consistent with the surface of the object. ### Claim 9 (depends on 7) The method of claim 7, further comprising: for each point in the point set, determining whether the point is visible with respect to each of the depth maps in the set; and, wherein the determination whether the point is photometrically consistent with the surface of the object is further based on whether the point is visible with respect to each of the depth maps in the set. ### Claim 10 (depends on 7) The method of claim 7, wherein the determination whether the point is photometrically consistent with the surface of the object based on the intersection points found is based on a standard deviation between color values of the point and the intersection points. ### Claim 11 A system for reconstructing a three-dimensional (3D) model of an object, the system comprising one or more of a processor configured to execute machine-readable instructions such that when the machine-readable instructions are executed by the processor, the processor is caused to perform: receiving images of the object, the images capturing the object from different viewpoints; obtaining a depth map for each of the images, the depth map including information relating to a surface of the object in a 3D space; projecting each pixel point in each depth map into the 3D space to obtain a point set; for each point in the point set, determining whether the point is consistent with the surface of the object in the 3D space as indicated by a set of depth maps for the images, wherein the consistency determination includes: for each depth map in the set: performing an intersection of a ray with the surface of the object indicated by the depth map to find an intersection point; and in response to the intersection point being found, determining a distance between the point and intersection point as a distance between the point and the surface of the object as seen from a camera viewpoint corresponding to the depth map; determining whether the point is consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object; and determining whether to discard the point from point set based on the determination whether the point is consistent with the surface of the object; and reconstructing the 3D model using the point set. ### Claim 12 (depends on 11) The system of claim 11, wherein the determination whether the point is consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes: averaging distances between the point and the surface of the object as indicated by the set of depth maps to obtain an average distance between the point and the surface of the object. ### Claim 13 (depends on 11) The system of claim 11, wherein the determination whether the point is consistent with the surface of the object is further based on a threshold distance value such that the point is determined to be geometrically consistent when the average distance between the point and the surface of the object is less than or equal to the threshold distance value. ### Claim 14 (depends on 11) The system of claim 11, wherein whether the point is consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes: truncating any distance between the point and the surface as indicated by a given depth map in the set to a preset value when the distance exceeds a threshold value. ### Claim 15 (depends on 11) The system of claim 11, wherein determining whether the point is consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes: assigning a weight to each of the determined distances, and determining the distance between the point and the surface of the object based on the weighted distances. ### Claim 16 (depends on 11) The system of claim 11, wherein determining whether the point is consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes: determining whether any distance between the point and the surface as indicated by a given depth map in the set is smaller than a threshold value indicating a negative distance value, and in response to determining the distance is smaller than the threshold value, excluding such a distance from the determination whether the point is geometrically consistent with the surface of the object. ### Claim 17 (depends on 11) The system of claim 11, wherein when executing the machine-readable instructions, the processor is further caused to perform: for each point in the point set, determining whether the point is photometrically consistent with the surface of the object based on the intersection points found; and, wherein the determination whether to discard the point from the point set is further based on whether the point is photometrically consistent the surface of the object. ### Claim 18 (depends on 17) The system of claim 17, wherein determining whether the point is photometrically consistent with the surface of the object based on the intersection points found includes: for each intersection point found: determining whether the distance between the point and intersection point is larger than a threshold value; and in response to determining the distance between the point and intersection point is larger than the threshold value, excluding the intersection point from the determination whether the point is photometrically consistent with the surface of the object. ### Claim 19 (depends on 17) The system of claim 17, wherein when executing the machine-readable instructions, the processor is further caused to perform: for each point in the point set, determining whether the point is visible with respect to each of the depth maps in the set; and, wherein the determination whether the point is photometrically consistent with the surface of the object is further based on whether the point is visible with respect to each of the depth maps in the set. ### Claim 20 (depends on 17) The system of claim 17, wherein the determination whether the point is photometrically consistent with the surface of the object based on the intersection points found is based on a standard deviation between color values of the point and the intersection points.