# US 10,484,561 B2 · Method and apparatus for scanning and printing a 3D object > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,484,561 B2 - Document ID: US10484561B2 - Status: Granted - Kind: B2 - Application number: 15/308,959 - Filed: 2015-05-11 - Published: 2017-03-23 - Granted: 2019-11-19 - Earliest claimed: 2014-05-12 - Assignees: ML Netherlands C.V. - Applicants: ML Netherlands C.V. - Inventors: Alexander Ilic; Benedikt Koeppel - Web page: https://patentgrep.com/patent/US10484561B2 ## CPC - H04N 1/00827: {Arrangements for reading an image from an unusual original, e.g. 3-dimensional objects} - B33Y 50/00: Data acquisition or data processing for additive manufacturing - G01B 11/24: for measuring contours or curvatures - G06T 7/55: from multiple images - G06T 7/74: {involving reference images or patches} - G06T 15/04: Texture mapping - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - H04N 13/211: using temporal multiplexing - H04N 13/221: using the relative movement between cameras and objects - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - H04N 13/275: from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals - G06T 2200/08: involving all processing steps from image acquisition to 3D model generation - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20076: Probabilistic image processing - G06T 2207/30244: Camera pose - G06T 2219/2021: Shape modification - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - B33Y: ADDITIVE MANUFACTURING, i.e. MANUFACTURING OF THREE-DIMENSIONAL [3D] OBJECTS BY ADDITIVE DEPOSITION, ADDITIVE AGGLOMERATION OR ADDITIVE LAYERING, e.g. BY 3D PRINTING, STEREOLITHOGRAPHY OR SELECTIVE LASER SINTERING - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A smartphone may be freely moved in three dimensions as it captures a stream of images of an object. Multiple image frames may be captured in different orientations and distances from the object and combined into a composite image representing an three-dimensional image of the object. The image frames may be formed into the composite image based on representing features of each image frame a set of points in a three dimensional depth map. Coordinates of the points in the depth map may be estimated with a level of certainty. The level of certainty may be used to determine which points are included in the composite image. The selected points may be smoothed and a mesh model may be formed by creating a convex hull of the selected points. The mesh model and associated texture information may be used to render a three-dimensional representation of the object on a two-dimensional display. Additional techniques include processing and formatting of the three-dimensional representation data to be printed by a three-dimensional printer so a three-dimensional model of the object may be formed. ## Claims ### Claim 1 A portable electronic device, comprising: a camera configured to acquire a plurality of image frames; at least one processor configured to, while the camera acquires the plurality of image frames: form a first representation of an object from the plurality of image frames acquired with the camera from a plurality of directions, the representation comprising locations in a three-dimensional space of features of the object; determine, from the first representation, a second representation of the object, the second representation comprising locations of one or more surfaces; automatically identify, from the second representation, surfaces that are not printable in three dimensions; provide information to a user of the camera instructing the user to capture additional image frames of a region of the object corresponding to the identified surfaces that are not printable in three dimensions, wherein providing the information includes: displaying the object with at least one colored frame to indicate the region of the object corresponding to the identified surfaces that are not printable in three dimensions; and outputting, to the user, position information indicating a position and an orientation of the camera to be used to capture the additional image frames, the position and the orientation of the camera determined to avoid creating image artifacts; and modify the second representation to remove at least one of the identified surfaces that are not printable in three dimensions, wherein the at least one processor is further configured to store the modified second representation as a three-dimensional printable file. ### Claim 2 (depends on 1) The portable electronic device of claim 1, wherein: modifying the second representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not joined to other surfaces to provide a wall thickness above a threshold. ### Claim 3 (depends on 1) The portable electronic device of claim 1, wherein: modifying the second representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not a part of a closed hull. ### Claim 4 (depends on 1) The portable electronic device of claim 1, wherein: modifying the second representation to remove surfaces that are not printable in three dimensions comprises computing normals to surfaces to remove surfaces having a normal in a direction toward an interior of a hull. ### Claim 5 (depends on 1) The portable electronic device of claim 1, wherein: the device further comprises one or more inertial sensors, and the at least one processor is further configured to form the first representation based on outputs of the one or more inertial sensors when the plurality of image frames is acquired. ### Claim 6 (depends on 1) The portable electronic device of claim 1, wherein: the device further comprises a display; and the at least one processor is further configured to render an image of the object based on a first portion of the plurality of image frames while a second portion of the plurality of image frames is being acquired. ### Claim 7 (depends on 1) The portable electronic device of claim 1, wherein: the three-dimensional printable file is in a portable document format. ### Claim 8 (depends on 1) The portable electronic device of claim 1, wherein: the three-dimensional printable file comprises separate information about structure of the object and visual characteristics of the structure. ### Claim 9 A method of forming a file for printing on a three-dimensional printer, the file comprising a representation of an object, the method comprising: acquiring a plurality of image frames using a camera of a portable electronic device; and while the image frames are being acquired: constructing a first three-dimensional representation of the object; determining a second three-dimensional representation of the object, by calculating a convex hull of the object; automatically identifying, from the second three-dimensional representation, areas that are not printable in three dimensions; providing information to a user of the camera instructing the user to capture additional image frames of a region of the object corresponding to the identified surfaces that are not printable in three dimensions, wherein providing the information includes: with the object on the display, depicting a boundary around at least a portion of the object, wherein the boundary indicates: the region of the object corresponding to the identified surfaces that are not printable in three dimensions; and position information indicating a position and an orientation of the camera to be used to capture the additional image frames, the position and the orientation of the camera determined to avoid creating image artifacts; adding at least one support structure to the second three-dimensional representation of the object that is not present in the first three-dimensional representation of the object; and displaying a view of the second three-dimensional representation on a two dimensional screen. ### Claim 10 (depends on 9) The method of claim 9, the method further comprises modifying the second three-dimensional representation to remove surfaces that are not printable in three dimensions. ### Claim 11 (depends on 10) The method of claim 10, wherein modifying the second three-dimensional representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not joined to other surfaces to provide a wall thickness above a threshold. ### Claim 12 (depends on 10) The method of claim 10, wherein modifying the second three-dimensional representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not a part of the convex hull. ### Claim 13 (depends on 10) The method of claim 10, wherein modifying the second three-dimensional representation to remove surfaces that are not printable in three dimensions comprises computing normals to surfaces to remove surfaces having a normal in a direction toward an interior of the convex hull. ### Claim 14 (depends on 9) The method of claim 9, wherein constructing the first representation comprises constructing the first representation based on outputs of one or more inertial sensors of the portable electronic device when the plurality of image frames is acquired. ### Claim 15 (depends on 9) The method of claim 9, the method further comprises rendering an image of the object based on a first portion of the plurality of image frames while a second portion of the plurality of image frames is being acquired. ### Claim 16 (depends on 9) The method of claim 9, wherein the three-dimensional printable file is in a portable document format. ### Claim 17 (depends on 9) The method of claim 9, wherein the three-dimensional printable file comprises separate information about structure of the object and visual characteristics of the structure. ### Claim 18 At least one non-transitory, tangible computer-readable storage medium having computer-executable instructions, that when executed by a processor, perform a method of forming a three dimensional representation of an object from a plurality of image frames captured with a camera of a portable electronic device, the method comprising: acquiring a plurality of image frames using a camera of a portable electronic device; while the image frames are being acquired: forming a first representation of the object from the plurality of image frames, the representation comprising locations in a three-dimensional space of features of the object; determining, from the first representation, a second representation of the object, the second representation comprising locations of one or more surfaces; and operating a processor so that it automatically identifies surfaces from the second representation that are not printable in three dimensions; operating the processor to provide information to a user of the camera instructing the user to capture additional image frames of a region of the object corresponding to the identified surfaces that are not printable in three dimensions, wherein providing the information includes: with the object on the display, depicting an at least one colored frame around at least a portion of the object to indicate the region of the object corresponding to the identified surfaces that are not printable in three dimensions; and outputting, to the user, position information indicating a position and an orientation of the camera to be used to capture the additional image frames, the position and the orientation of the camera determined to avoid creating image artifacts; and operating the processor to modify the second representation to remove at least one of the identified surfaces that are not printable in three dimensions; and storing the modified second representation as a three-dimensional printable file. ### Claim 19 (depends on 18) The at least one computer-readable storage medium of claim 18, wherein modifying the second representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not joined to other surfaces to provide a wall thickness above a threshold. ### Claim 20 (depends on 18) The at least one computer-readable storage medium of claim 18, wherein modifying the second representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not a part of a closed hull. ### Claim 21 (depends on 18) The at least one computer-readable storage medium of claim 18, wherein modifying the second representation to remove surfaces that are not printable in three dimensions comprises computing normals to surfaces to remove surfaces having a normal in a direction toward an interior of a hull. ### Claim 22 (depends on 18) The at least one computer-readable storage medium of claim 18, wherein forming the first representation comprises forming the first representation based on outputs of one or more inertial sensors of the portable electronic device when the plurality of image frames is acquired. ### Claim 23 (depends on 18) The at least one computer-readable storage medium of claim 18, wherein the method further comprises rendering an image of the object based on a first portion of the plurality of image frames while a second portion of the plurality of image frames is being acquired. ### Claim 24 (depends on 18) The at least one computer-readable storage medium of claim 18, wherein the three-dimensional printable file is in a portable document format. ### Claim 25 (depends on 18) The at least one computer-readable storage medium of claim 18, wherein the three-dimensional printable file comprises separate information about structure of the object and visual characteristics of the structure.