# US 2018/0025496 A1 · Systems and methods for improved surface normal estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0025496 A1 - Document ID: US20180025496A1 - Status: Published application - Kind: A1 - Application number: 15/214,141 - Filed: 2016-07-19 - Published: 2018-01-25 - Earliest claimed: 2016-07-19 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Albrecht Johannes Lindner; Kalin Mitkov Atanassov; James Wilson Nash - Web page: https://patentgrep.com/patent/US20180025496A1 ## CPC - G06T 15/04: Texture mapping - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - G06T 7/50: Depth or shape recovery - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 2200/04: involving 3D image data - G06T 2200/08: involving all processing steps from image acquisition to 3D model generation - G06T 2207/10012: Stereo images - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2210/56: Particle system, point based geometry or rendering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Methods and systems for surface normal estimation are disclosed. In some aspects, a plurality of images or depth maps representing a three dimensional object from multiple viewpoints is received. Surface normals at surface points within a single image of the plurality of images are estimated based on surface points within the single image. An electronic representation of a three dimensional surface of the object is generated based on the surface normals and a point cloud comprised of surface points derived from the plurality of images. ## Claims ### Claim 1 A method of electronically representing a three dimensional surface for an object, comprising: receiving a plurality of images of the object, the images representing the object from a plurality of viewpoints; deriving points on the surface of the object in three dimensional space from each of the plurality of images; generating estimates of surface normals at surface points within each of the plurality of images based on the surface points derived from the image; generating a point cloud comprising surface points derived from each of the plurality of images; and generating a representation of the three dimensional surface for the object based on the surface normal estimates and the point cloud. ### Claim 2 (depends on 1) The method of claim 1, further comprising rendering the representation of the three dimensional surface on an electronic display or storing the representation to a stable storage device. ### Claim 3 (depends on 1) The method of claim 1, further comprising determining a number of surface points derived from a first image that are also within a neighborhood of a particular surface point derived from the first image; generating an estimate of a surface normal at the particular surface point based on the surface points derived from the first image in response to the number of surface points within the neighborhood exceeding a threshold; generating the estimate of the surface normal at the particular surface point based on surface points derived from a second image of the plurality of images in response to the number of surface points within the neighborhood being below the threshold; and generating the representation based on the estimate. ### Claim 4 (depends on 1) The method of claim 1, further comprising: determining first surface points from a first image of the plurality of images; determining first estimates of surface normals for at least a portion of the first surface points; determining a consistency between the first estimates; in response to the consistency being below a consistency threshold: generating second estimates of the surface normals based on second surface points derived from a second image of the plurality of images, and generating the representation based on the second estimates. ### Claim 5 (depends on 4) The method of claim 4, wherein determining the consistency between the first estimates is based on an inner product of at least two of the surface normals in the first estimates. ### Claim 6 (depends on 1) The method of claim 1, wherein the plurality of images comprises a plurality of corresponding depth maps, the depth maps indicating distances from the surface of the object from the plurality of viewpoints. ### Claim 7 (depends on 6) The method of claim 6, wherein deriving surface points in three dimensional space is based on the distances. ### Claim 8 An apparatus for electronically representing a three dimensional surface for an object, comprising: an electronic hardware processor; an electronic hardware memory, storing instructions that when executed, cause the electronic hardware processor to: receive a plurality of images of the object, the images representing the object from a plurality of viewpoints; derive points on the surface of the object in three dimensional space from each of the plurality of images; generate estimates of surface normals at surface points within each of the plurality of images based on the surface points derived from the image; generate a point cloud comprising surface points derived from each of the plurality of images; and generate a representation of the three dimensional surface for the object based on the surface normal estimates and the point cloud. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the electronic hardware memory stores further instructions that cause the electronic hardware processor to render the representation of the three dimensional surface on an electronic display or store the representation to a stable storage device. ### Claim 10 (depends on 8) The apparatus of claim 8, wherein the electronic hardware memory stores further instructions that cause the electronic hardware processor to: determine a number of surface points derived from a first image that are also within a neighborhood of a particular surface point derived from the first image, generate an estimate of a surface normal at the particular surface point based on the surface points derived from the first image in response to the number of surface points within the neighborhood exceeding a threshold, generate the estimate of the surface normal at the particular surface point based on surface points derived from a second image of the plurality of images in response to the number of surface points within the neighborhood being below the threshold, and generate the representation based on the estimate. ### Claim 11 (depends on 8) The apparatus of claim 8, wherein the electronic hardware memory stores further instructions that cause the electronic hardware processor to: determine first surface points from a first image of the plurality of images, determine first estimates of surface normals for at least a portion of the first surface points, determine a consistency between the first estimates, in response to the consistency being below a consistency threshold: generate second estimates of the surface normals based on second surface points derived from a second image of the plurality of images, and generate the representation based on the second estimates. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein the electronic hardware memory stores further instructions that cause the electronic hardware processor to determine the consistency between the first estimates based on an inner product of at least two of the surface normals in the first estimates. ### Claim 13 (depends on 8) The apparatus of claim 8, wherein the plurality of images comprises a plurality of corresponding depth maps, the plurality of depth maps indicating distances from the surface of the object to the plurality of viewpoints, wherein the deriving of surface points in three dimensional space is based on the distances. ### Claim 14 A computer readable storage medium comprising instructions that when executed cause an electronic hardware processor to perform a method of electronically representing a three dimensional surface for an object, the method comprising: receiving a plurality of images of the object, the images representing the object from a plurality of viewpoints; deriving points on the surface of the object in three dimensional space from each of the plurality of images; generating estimates of surface normals at surface points within each of the plurality of images based on the surface points derived from the image; generating a point cloud comprising surface points derived from each of the plurality of images; and generating a representation of the three dimensional surface for the object based on the surface normal estimates and the point cloud. ### Claim 15 (depends on 14) The computer readable storage medium of claim 14, the method further comprising: determining a number of surface points derived from a first image that are also within a neighborhood of a particular surface point derived from the first image; generating an estimate of a surface normal at the particular surface point based on the surface points derived from the first image in response to the number of surface points within the neighborhood exceeding a threshold; generating the estimate of the surface normal at the particular surface point based on surface points derived from a second image of the plurality of images in response to the number of surface points within the neighborhood being below the threshold; and generating the representation based on the estimate. ### Claim 16 (depends on 15) The computer readable storage medium of claim 15, the method further comprising: determining first surface points from a first image of the plurality of images; determining first estimates of surface normals for at least a portion of the first surface points; determining a consistency between the first estimates; in response to the consistency being below a consistency threshold: generating second estimates of the surface normals based on second surface points derived from a second image of the plurality of images, and generating the representation based on the second estimates. ### Claim 17 (depends on 16) The computer readable storage medium of claim 16, wherein determining the consistency between the first estimates is based on an inner product of at least two of the surface normals in the first estimates. ### Claim 18 (depends on 15) The computer readable storage medium of claim 15, wherein the plurality of images comprises a plurality of corresponding depth maps, the depth maps indicating distances from the surface of the object from the plurality of viewpoints, wherein the deriving of points on the surface of the object in three dimensional space from each of the plurality of images is based on the indicated distances. ### Claim 19 A system for three dimensional object reconstruction, comprising: a depth sensor; an imaging sensor; an electronic display; an electronic hardware processor; an electronic hardware memory, storing instructions that when executed, cause the electronic hardware processor to: receive a plurality of images of a three dimensional object from the imaging sensor, the images representing the three dimensional object from a plurality of viewpoints; derive points on the surface of the three dimensional object in three dimensional space from each of the plurality of images and based on input from the depth sensor; generate estimates of surface normals at surface points within each of the plurality of images based on the surface points derived from the image; generate a point cloud comprising surface points derived from each of the plurality of images; generate a representation of the three dimensional surface for the object based on the surface normal estimates and the point cloud; and display the representation on the electronic display. ### Claim 20 (depends on 19) The system of claim 19, wherein the electronic hardware memory stores further instructions that cause the electronic hardware processor to render the representation of the three dimensional surface on the electronic display or store the representation to a stable storage device. ### Claim 21 (depends on 19) The system of claim 19, wherein the electronic hardware memory stores further instructions that cause the electronic hardware processor to: determine a number of surface points derived from a first image that are also within a neighborhood of a particular surface point derived from the first image; generate an estimate of a surface normal at the particular surface point based on the surface points derived from the first image in response to the number of surface points within the neighborhood exceeding a threshold; generate the estimate of the surface normal at the particular surface point based on surface points derived from a second image of the plurality of images in response to the number of surface points within the neighborhood being below the threshold; and generate the representation based on the estimate. ### Claim 22 (depends on 19) The system of claim 19, wherein the electronic hardware memory stores further instructions that cause the electronic hardware processor to: determine first surface points from a first image of the plurality of images; determine first estimates of surface normals for at least a portion of the first surface points; determine a consistency between the first estimates; in response to the consistency being below a consistency threshold: generate second estimates of the surface normals based on second surface points derived from a second image of the plurality of images, and generate the representation based on the second estimates. ### Claim 23 (depends on 22) The system of claim 22, wherein the electronic hardware memory stores further instructions that cause the electronic hardware processor to determine the consistency between the first estimates based on an inner product of at least two of the surface normals in the first estimates. ### Claim 24 (depends on 19) The system of claim 19, wherein the plurality of images comprises a plurality of corresponding depth maps, the plurality of depth maps indicating distances from the surface of the object to the plurality of viewpoints, wherein the deriving of the surface points in three dimensional space is based on the distances.