# US 10,311,648 B2 · Systems and methods for scanning three-dimensional objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,311,648 B2
- Document ID: US10311648B2
- Status: Granted
- Kind: B2
- Application number: 15/630,715
- Filed: 2017-06-22
- Published: 2017-12-28
- Granted: 2019-06-04
- Earliest claimed: 2016-06-22
- Assignees: Aquifi, Inc.
- Applicants: Aquifi, Inc.
- Inventors: Giridhar Murali; Keith Blackstone; Francesco Peruch; Carlo Dal Mutto
- Web page: https://patentgrep.com/patent/US10311648B2

## CPC

- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G01S 7/4817: {relating to scanning}
- G01S 17/00: Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S 17/48: Active triangulation systems, i.e. using the transmission and reflection of electromagnetic waves other than radio waves
- G01S 17/87: Combinations of systems using electromagnetic waves other than radio waves
- G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06T 7/337: {involving reference images or patches}
- G06T 7/38: Registration of image sequences
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 13/296: Synchronisation thereof; Control thereof
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20221: Image fusion; Image merging
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T 2219/2016: Rotation, translation, scaling
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for computing a three-dimensional (3D) model of an object includes: receiving, by a processor, a first chunk including a 3D model of a first portion of the object, the first chunk being generated from a plurality of depth images of the first portion of the object; receiving, by the processor, a second chunk including a 3D model of a second portion the object, the second chunk being generated from a plurality of depth images of the second portion of the object; computing, by the processor, a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks; aligning, by the processor, the first chunk with the second chunk in accordance with the registration; and outputting, by the processor, a 3D model corresponding to the first chunk merged with the second chunk.

## Claims

### Claim 1

A method for computing a three-dimensional (3D) model of an object, the method comprising:
receiving, by a processor, a first chunk comprising a 3D model of a first portion of the object, the first chunk being generated from a plurality of depth images of the first portion of the object;
receiving, by the processor, a second chunk comprising a 3D model of a second portion the object, the second chunk being generated from a plurality of depth images of the second portion of the object;
computing, by the processor, a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks, the computing the registration of the first chunk with the second chunk comprising:
extracting, by the processor, first key points from color textures of a first surface of the first chunk;
extracting, by the processor, second key points from color textures of a second surface of the second chunk; and
computing a rigid transformation from the second key points to the first key points;
aligning, by the processor, the first chunk with the second chunk in accordance with the registration; and
outputting, by the processor, a 3D model corresponding to the first chunk merged with the second chunk.

### Claim 2 (depends on 1)

The method of claim 1, wherein the computing the registration of the first chunk with the second chunk comprises:
computing a first cuboid bounding box of the first chunk;
computing a second cuboid bounding box of the second chunk;
finding a plurality of rigid transformations of the first cuboid onto the second cuboid;
computing a plurality of registration costs, each of the registration costs corresponding to one of the rigid transformations; and
selecting a rigid transformation of the rigid transformations in accordance with the registration costs, the registration comprising the rigid transformation of the rigid transformations.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
computing a registration cost of the rigid transformation;
modifying the registration cost in accordance with inconsistency between matching points of the first chunk and the second chunk; and
computing the registration in accordance with a rigid transformation having a minimized registration cost.

### Claim 4

A method for computing a three-dimensional (3D) model of an object, the method comprising:
receiving, by a processor, a first chunk comprising a 3D model of a first portion of the object, the first chunk being generated from a plurality of depth images of the first portion of the object;
receiving, by the processor, a second chunk comprising a 3D model of a second portion the object, the second chunk being generated from a plurality of depth images of the second portion of the object;
computing, by the processor, a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks, the computing the registration of the first chunk with the second chunk comprising:
extracting, by the processor, first key points from color textures of a first surface of the first chunk;
extracting, by the processor, second key points from color textures of a second surface of the second chunk; and
computing a rigid transformation from the second key points to the first key points;
aligning, by the processor, the first chunk with the second chunk in accordance with the registration; and
outputting, by the processor, a 3D model corresponding to the first chunk merged with the second chunk,
wherein the computing the registration of the first chunk with the second chunk further comprises:
receiving a user supplied transformation of the first chunk, the user supplied transformation comprising at least one transformation of the group comprising:
a rotation;
a translation; and
a zoom; and
applying the user supplied transformation to the first chunk.

### Claim 5 (depends on 4)

The method of claim 4, wherein the computing the registration of the first chunk with the second chunk further comprises:
computing a first cuboid bounding box of the first chunk;
computing a second cuboid bounding box of the second chunk;
identifying three vertices of the first chunk;
transforming the three vertices of the first cuboid in accordance with the user supplied transformation;
identifying the three vertices of the second cuboid corresponding to the transformed three vertices of the first cuboid;
computing a vertex transformation of the three vertices of the first cuboid to the three corresponding vertices of the second cuboid;
initializing the transformation aligning corresponding portions of the first and second chunks in accordance with the user supplied transformation and the vertex transformation; and
refining the transformation aligning corresponding portions of the first and second chunks in accordance with a local registration method.

### Claim 6 (depends on 1)

The method of claim 1, wherein the aligning the first chunk with the second chunk in accordance with the registration comprises:
transforming the first chunk in accordance with the registration;
combining matching points between the first chunk and the second chunk; and
adding unique points from the first chunk and the second chunk.

### Claim 7 (depends on 1)

The method of claim 1, wherein the method further comprises:
when the registration of the first chunk with the second chunk fails:
receiving an additional depth frame corresponding to the second chunk;
updating the 3D model of the second chunk in accordance with the additional depth frame to compute an updated second chunk; and
computing the registration of the first chunk with the updated second chunk.

### Claim 8 (depends on 1)

The method of claim 1, wherein the 3D model of the first portion of the object corresponds to the object in a first configuration, and the 3D model of the second portion of the object corresponds to the object in a second configuration.

### Claim 9

A method for computing a three-dimensional (3D) model of an object, the method comprising:
receiving, by a processor, a first chunk comprising a 3D model of a first portion of the object, the first chunk being generated from a plurality of depth images of the first portion of the object;
receiving, by the processor, a second chunk comprising a 3D model of a second portion the object, the second chunk being generated from a plurality of depth images of the second portion of the object;
computing, by the processor, a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks;
aligning, by the processor, the first chunk with the second chunk in accordance with the registration;
outputting, by the processor, a 3D model corresponding to the first chunk merged with the second chunk;
detecting portions of the depth images of the second portion of the object corresponding to human hands;
removing the portions of the depth images corresponding to the human hands; and
generating the second chunk from the depth images with the portions corresponding to human hands removed from the depth images.

### Claim 10

A system for generating a three-dimensional (3D) scan of an object, the system comprising:
a depth camera system comprising a sensor;
a display device;
a processor coupled to depth camera system and the display device; and
memory storing instructions that, when executed by the processor, cause the processor to:
control the depth camera system to capture a first plurality of depth images;
compute a first chunk comprising a 3D model of a first portion of the object generated from the first plurality of depth images;
control the depth camera system to capture a second plurality of depth images;
compute a second chunk comprising a 3D model of a second portion of the object generated from the second plurality of depth images;
compute a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks by:
extracting, by the processor, first key points from color textures of a first surface of the first chunk;
extracting, by the processor, second key points from color textures of a second surface of the second chunk; and
computing a rigid transformation from the second key points to the first key points;
align the first chunk with the second chunk in accordance with the registration; and
display, on the display device, a 3D model corresponding to the first chunk merged with the second chunk.

### Claim 11 (depends on 10)

The system of claim 10, wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the registration of the first chunk with the second chunk by:
computing, by the processor, a first cuboid bounding box of the first chunk;
computing, by the processor, a second cuboid bounding box of the second chunk;
finding, by the processor, a plurality of rigid transformations of the first cuboid onto the second cuboid;
computing, by the processor, a plurality of registration costs, each of the registration costs corresponding to one of the rigid transformations; and
selecting, by the processor, a rigid transformation of the rigid transformations in accordance with the registration costs, the registration comprising the rigid transformation of the rigid transformations.

### Claim 12 (depends on 10)

The system of claim 10, wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the registration of the first chunk with the second chunk by:
computing a registration cost of the rigid transformation;
modifying the registration cost in accordance with inconsistency between matching points of the first chunk and the second chunk; and
computing the registration in accordance with a rigid transformation having a minimized registration cost.

### Claim 13

A system for generating a three-dimensional (3D) scan of an object, the system comprising:
a depth camera system comprising a sensor;
a display device;
a processor coupled to depth camera system and the display device; and
memory storing instructions that, when executed by the processor, cause the processor to:
control the depth camera system to capture a first plurality of depth images;
compute a first chunk comprising a 3D model of a first portion of the object generated from the first plurality of depth images;
control the depth camera system to capture a second plurality of depth images;
compute a second chunk comprising a 3D model of a second portion of the object generated from the second plurality of depth images;
receive a user supplied transformation of the first chunk, the user supplied transformation comprising at least one transformation of the group comprising:
a rotation;
a translation; and
a zoom; and
apply the user supplied transformation to the first chunk;
compute a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks, wherein the instructions to compute the registration of the first chunk with the second chunk comprise instructions that, when executed by the processor, cause the processor to:
extract first key points from color textures of a first surface of the first chunk;
extract second key points from color textures of a second surface of the second chunk; and
compute a rigid transformation from the second key points to the first key points;
align the first chunk with the second chunk in accordance with the registration; and
display, on the display device, a 3D model corresponding to the first chunk merged with the second chunk.

### Claim 14 (depends on 13)

The system of claim 13, wherein the display device is a touchscreen, and
wherein the user supplied transformation is received via a touch input supplied to the touchscreen.

### Claim 15 (depends on 13)

The system of claim 13, wherein the user supplied transformation is detected as a three dimensional gesture detected by the depth camera system.

### Claim 16 (depends on 13)

The system of claim 13, wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the registration of the first chunk with the second chunk by:
computing a first cuboid bounding box of the first chunk;
computing a second cuboid bounding box of the second chunk;
identifying three vertices of the first chunk;
transforming the three vertices of the first cuboid in accordance with the user supplied transformation;
identifying the three vertices of the second cuboid corresponding to the transformed three vertices of the first cuboid;
computing a vertex transformation of the three vertices of the first cuboid to the three corresponding vertices of the second cuboid;
initializing the transformation aligning corresponding portions of the first and second chunks in accordance with the user supplied transformation and the vertex transformation; and
refining the transformation aligning corresponding portions of the first and second chunks in accordance with a local registration method.

### Claim 17 (depends on 10)

The system of claim 10, wherein the memory further stores instructions that, when executed by the processor, cause the processor to align the first chunk with the second chunk by:
transforming the first chunk in accordance with the registration;
combining matching points between the first chunk and the second chunk; and
adding unique points from the first chunk and the second chunk.

### Claim 18 (depends on 10)

The system of claim 10, wherein the memory further stores instructions that, when executed by the processor, cause the processor to, when the registration of the first chunk with the second chunk fails:
receive an additional depth frame corresponding to the second chunk;
update the 3D model of the second chunk in accordance with the additional depth frame to compute an updated second chunk; and
compute the registration of the first chunk with the updated second chunk.

### Claim 19 (depends on 10)

The system of claim 10, wherein the 3D model of the first portion of the object corresponds to the object in a first configuration, and the 3D model of the second portion of the object corresponds to the object in a second configuration.

### Claim 20

A system for generating a three-dimensional (3D) scan of an object, the system comprising:
a depth camera system comprising a sensor;
a display device;
a processor coupled to depth camera system and the display device; and
memory storing instructions that, when executed by the processor, cause the processor to:
control the depth camera system to capture a first plurality of depth images;
compute a first chunk comprising a 3D model of a first portion of the object generated from the first plurality of depth images;
control the depth camera system to capture a second plurality of depth images;
compute a second chunk comprising a 3D model of a second portion of the object generated from the second plurality of depth images;
compute a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks;
align the first chunk with the second chunk in accordance with the registration;
display, on the display device, a 3D model corresponding to the first chunk merged with the second chunk
detect portions of the depth images of the second portion of the object corresponding to human hands;
remove the portions of the depth images corresponding to the human hands; and
generate the second chunk from the depth images with the portions corresponding to human hands removed from the depth images.

### Claim 21 (depends on 10)

The system of claim 10, wherein the display device is integrated into a pair of augmented reality glasses.

### Claim 22 (depends on 21)

The system of claim 21, wherein the depth camera system is integrated into the pair of augmented reality glasses.

### Claim 23

A method for assembling a plurality of chunks corresponding to separate three-dimensional (3D) models of different portions of an object, the chunks being displayed in a graphical user interface, the method comprising:
receiving, by a processor, a selection of a first chunk of the plurality of chunks, the first chunk comprising a 3D model of a first portion of the object;
receiving, by the processor, a selection of a second chunk of the plurality of chunks, the second chunk comprising a 3D model of a second portion of the object;
computing, by the processor, a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks, the computing the registration of the first chunk with the second chunk comprising:
extracting, by the processor, first key points from color textures of a first surface of the first chunk;
extracting, by the processor, second key points from color textures of a second surface of the second chunk; and
computing a rigid transformation from the second key points to the first key points;
aligning, by the processor, the first chunk with the second chunk in accordance with the registration; and
displaying, by the processor, the first chunk assembled with the second chunk as a chunk assembly.

### Claim 24 (depends on 23)

The method of claim 23, further comprising:
receiving user input to transform a view of the chunk assembly, the user input comprising at least one transformation of the group comprising:
a rotation;
a translation; and
a zoom.

### Claim 25 (depends on 23)

The method of claim 23, further comprising:
receiving a user input to transform the first chunk, the user input comprising at least one transformation of the group comprising a rotation and a translation; and
updating the displaying of the first chunk in accordance with the user input.

### Claim 26 (depends on 23)

The method of claim 23, further comprising displaying a confidence of the chunk assembly.

### Claim 27 (depends on 23)

The method of claim 23, wherein points in the chunk assembly are colored in accordance with whether the points originated from the first chunk or the second chunk.

### Claim 28 (depends on 23)

The method of claim 23, wherein the plurality of chunks are associated with a scanning project, the scanning project comprising one or more versions of assemblies of one or more chunks, each version being associated with metadata storing a collection of chunks that are included an assembly associated with the version.

### Claim 29 (depends on 23)

The method of claim 23, wherein at least one chunk of the plurality of chunks is received from a remote source.