# US 10,482,679 B2 · Capturing and aligning three-dimensional scenes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,679 B2 - Document ID: US10482679B2 - Status: Granted - Kind: B2 - Application number: 14/070,426 - Filed: 2013-11-01 - Published: 2014-05-08 - Granted: 2019-11-19 - Earliest claimed: 2012-02-24 - Assignees: Matterport, Inc. - Applicants: Matterport, Inc. - Inventors: Matthew Bell; David Gausebeck; Michael Beebe - Web page: https://patentgrep.com/patent/US10482679B2 ## CPC - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06V 20/653: {by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces} - H04N 13/10: Processing, recording or transmission of stereoscopic or multi-view image signals - H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597) - H04N 13/156: Mixing image signals - H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00) - H04N 13/246: Calibration of cameras - H04N 2213/001: Constructional or mechanical details - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems and methods for building a three-dimensional composite scene are disclosed. Certain embodiments of the systems and methods may include the use of a three-dimensional capture device that captures a plurality of three-dimensional images of an environment. Some embodiments may further include elements concerning aligning and/or mapping the captured images. Various embodiments may further include elements concerning reconstructing the environment from which the images were captured. The methods disclosed herein may be performed by a program embodied on a non-transitory computer-readable storage medium when executed the program is executed a processor. ## Claims ### Claim 1 A three-dimensional capture system, comprising: a rotatable stage configured to rotate about an axis; one or more three-dimensional capture devices mounted on the rotatable stage and configured to capture, during a first rotation of the rotatable stage at a first location in an environment, first images of the environment; and a communication device configured to send the first images to a computation device as they are respectively captured, wherein the computation device is configured to align the first images to one another using a pair-wise alignment process based on correspondences between position information determined for features included in respective first images of the first images, resulting in a first set of aligned image data captured at the first location, wherein the position information is determined based on depth information respectively associated with the features, and wherein the computation device is further configured to generate a three-dimensional reconstruction of the environment by aligning the first set of aligned image data with a second set of aligned image data using a global alignment process, wherein the second set of aligned image data comprises second images that were captured during a second rotation of the rotatable stage at a second location in the environment and aligned to one another using the pair-wise alignment process, and wherein the global alignment process comprises aligning the first set of aligned image data with the second set of aligned imaged data in a common three-dimensional coordinate space. ### Claim 2 (depends on 1) The three-dimensional capture system of claim 1, further comprising a fixed frame provided on the rotatable stage, wherein the computation device and the one or more three-dimensional captured devices are configured to mount on the rotatable staged via the fixed frame. ### Claim 3 (depends on 1) The three-dimensional capture system of claim 1, wherein the communication device comprises a docking station that receives the computation device and provides for at least one of, physical or electrical coupling between the computation device and the one or more three-dimensional capture devices and the rotatable stage. ### Claim 4 (depends on 3) The three-dimensional capture system of claim 3, wherein the computation device is at least one of: a mobile phone, a tablet computer, a laptop computer, or a wearable computer. ### Claim 5 (depends on 1) The three-dimensional capture system of claim 1, wherein the rotatable stage comprises a motor and wherein the motor is configured to control rotation of the rotatable stage in response to reception of control signals from the computation device via a wired or wireless connection between the computation device and the motor. ### Claim 6 (depends on 1) The three-dimensional capture system of claim 1, wherein the one or more three-dimensional capture devices comprise three-dimensional sensors configured to capture the first images and the second images, and wherein the three-dimensional sensors are respectively positioned at different angles relative to a vertical plane such that respective fields of view of adjacent sensors overlap. ### Claim 7 (depends on 1) The three-dimensional capture system of claim 1, wherein the computation device is further configured to render the three-dimensional reconstruction of the environment via a display. ### Claim 8 (depends on 1) The three-dimensional capture system of claim 1, wherein the computation device is configured to align the first images to one another as they are received during the first rotation of the rotatable stage. ### Claim 9 A three-dimensional capture system, comprising: a rotatable stage that has mounted thereon, one or more three-dimensional capture devices configured to capture first images at various rotation points during a first rotation of the rotatable stage at a first location in an environment; a memory that stores computer executable instructions; and a processor, coupled to the memory, that facilitates execution of the executable instructions to perform operations, comprising: aligning the first images to one another using a pair-wise alignment process based on correspondences between position information determined for features included in respective first images of the first images, thereby generating a first set of aligned image data captured at the first location, wherein the position information is determined based on depth information respectively associated with the features; and sending the first set of aligned image data to a computation device, wherein based on the sending, the computation device is configured to perform a global alignment process that comprises aligning the first set of aligned image data with a second set of aligned image data in a common three-dimensional coordinate space to generate a three-dimensional reconstruction of the environment, wherein the second set of aligned image data comprises second images that were captured during a second rotation of the rotatable stage at a second location in the environment and aligned to one another using the pair-wise alignment process. ### Claim 10 (depends on 9) The three-dimensional capture system of claim 9, wherein the operations further comprise: receiving rotation information identifying respective rotation points of the rotatable stage associated with capture of the respective first images, and wherein the performing the pair-wise alignment process comprises aligning the first images to one another during the first rotation of the rotatable stage based on the rotation information. ### Claim 11 (depends on 9) The three-dimensional capture system of claim 9, further comprising a motor configured to drive rotation of the rotatable stage, and wherein the operations further comprise: receiving, from a remote device, control signals requesting rotation of the rotatable stage; and controlling the first and the second rotation of the rotatable stage via the motor based on the control signals. ### Claim 12 (depends on 9) The three-dimensional capture system of claim 9, further comprising a robotic mount configured to move positions and a motor configured to drive movement of the robotic mount, wherein the rotatable stage is mounted on the robotic mount and wherein the operations further comprise: receiving a control signal requesting movement of the robotic mount to reposition the one or more three-dimensional capture devices relative to an area of the environment in response to a determination, based on the first aligned image data or the three-dimensional reconstruction, that the area of the environment is associated with low-quality image data; and controlling movement of the robotic mount via the motor based on the control signal. ### Claim 13 (depends on 9) The three-dimensional capture system of claim 9, further comprising a robotic mount configured to move positions and a motor configured to drive movement of the robotic mount, wherein the rotatable stage is mounted on the robotic mount and wherein the operations further comprise: receiving, from the computation device, information identifying an area of the environment associated with no or low-quality image data as determined based on analysis of the three-dimensional reconstruction by the computation device; controlling movement of the robotic mount to reposition the one or more three-dimensional capture devices relative to the area of the environment; and capturing one or more new images of the area via the one or more three-dimensional capture devices. ### Claim 14 (depends on 9) The three-dimensional capture system of claim 9, wherein the one or more three-dimensional capture devices comprise at least two three-dimensional capture devices and wherein the at least two three-dimensional capture devices are calibrated such that image data captured by the at least two three-dimensional capture devices can be aligned onto the common three-dimensional coordinate space. ### Claim 15 (depends on 9) The three-dimensional capture system of claim 9, wherein the global alignment process comprises a graph optimization process. ### Claim 16 A method comprising: controlling, by a first device comprising a processor and memory, rotation of a rotatable stage comprising a plurality of three-dimensional capture devices mounted thereon; directing, by the first device, the plurality of three-dimensional capture devices to capture first images of an environment at different rotation points during a first rotation of the rotatable stage at a first location in the environment, wherein the first images are respectively associated with distance information for features respectively included therein; receiving, by the first device, the first images as they are captured during the first rotation; aligning, by the first device, the images to one another using a pair-wise alignment based on correspondences between position information determined for features included in respective first images of the first images, thereby generating a first set of aligned image data captured at the first location, wherein the position information is determined based on distance information respectively associated with the features; and sending, by the first device, the first set of aligned image data to a second device, wherein based on the sending, the second device is configured to generate a three-dimensional reconstruction of the environment by aligning the first set of aligned image data with a second set of aligned image data in a common three-dimensional coordinate space using a global alignment process, wherein the second set of aligned image data comprises second images that were captured during a second rotation of the rotatable stage at a second location in the environment and aligned to one another using the pair-wise alignment process. ### Claim 17 (depends on 16) The method of claim 16, further comprising: receiving rotation information identifying respective rotation points of the rotatable stage associated with capture of the respective first images, and wherein the aligning the first images to one another further comprises aligning the first images as they are received based on the rotation information. ### Claim 18 A non-transitory computer-readable storage medium comprising computer-readable instructions that, in response to execution, cause a computing system to perform operations, comprising: controlling rotation of a three-dimensional capture device as the three-dimensional capture device captures first images of an environment at various rotation points relative to a first fixed location in the environment; receiving the first images as they are respectively captured; determining three-dimensional position information for points included in respective first images of the first images based on respective capture distances between the points and the three-dimensional capture device at times of capture of the respective first images; aligning the first images to one another using a pair-wise alignment process based on correspondences between the three-dimensional position information for the points, thereby generating a first set of aligned image data captured at the first fixed location; and generating a three-dimensional representation of the environment by further aligning the first set of aligned image data with a second set of aligned image data using a global alignment process, wherein the second set of aligned image data comprises second images that were captured during a second rotation of the rotatable stage at a second fixed location in the environment and aligned to one another using the pair-wise alignment process, and wherein the global alignment process comprises aligning the first set of aligned image data with the second set of aligned imaged data in a common three-dimensional coordinate space. ### Claim 19 (depends on 18) The non-transitory computer-readable storage medium of claim 18, the operations further comprising: rendering the three-dimensional representation of the environment via a display as it is generated. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium of claim 19, the operations further comprising: receiving input identifying a perspective for viewing the three-dimensional representation of the environment as rendered via the display; and rendering the three-dimensional representation of the environment in accordance with the perspective. ### Claim 21 (depends on 1) The three-dimensional capture system of claim 1, wherein the computation device is further configured to receive rotation information identifying respective rotation points of the rotatable stage associated with capture of the respective images, and wherein the computation device is further configured to align the first images to one another based on the rotation information. ### Claim 22 (depends on 1) The three-dimensional capture system of claim 1, wherein the position information is further determined based on respective capture positions and orientations of the first images. ### Claim 23 (depends on 1) The three-dimensional capture system of claim 1, wherein the computation device is further configured to align the first images to one another based on correspondences in visual characteristics of the features. ### Claim 24 (depends on 23) The three-dimensional capture system of claim 23, wherein the features include pixels and wherein the visual characteristics include color or brightness of the pixels. ### Claim 25 (depends on 1) The three-dimensional capture system of claim 1, wherein the global alignment process comprises a graph optimization process. ### Claim 26 (depends on 1) The three-dimensional capture system of claim 1, wherein the computation device is further configured to align the first images to one another using a scoring mechanism that scores potential alignments between the respective images based on the correspondences between the position information for the features included in the respective images. ### Claim 27 (depends on 1) The three-dimensional capture system of claim 1, wherein the first set of aligned image data corresponds to a panoramic view of the environment at the first location. ### Claim 28 (depends on 1) The three-dimensional capture system of claim 1, wherein the second set of aligned image data corresponds to a panoramic view of the environment at the second location. ### Claim 29 (depends on 9) The three-dimensional capture system of claim 9, further comprising a motor configured to drive rotation of the rotatable stage, and wherein the operations further comprise: identifying, based on the first set of aligned image data, an area of the environment associated with no or low-quality image data; controlling rotation of the rotatable stage via the motor to position the one or more three-dimensional capture devices relative to the area; directing the one or more three-dimensional capture devices to capture one or more additional images of the area; and re-aligning the first images to one another the first alignment process again using the images and the one or more additional images to generate an updated first set of the aligned image data. ### Claim 30 (depends on 9) The three-dimensional capture system of claim 9, further comprising a robotic mount configured to move positions and a motor configured to drive movement of the robotic mount, wherein the rotatable stage is mounted on the robotic mount and wherein the operations further comprise: determining, based on the first aligned image data, an area of the environment is associated with no or low-quality image data; controlling movement of the robotic mount via the motor to position the one or more three-dimensional capture devices relative to the area; directing the one or more three-dimensional capture devices to capture one or more additional images of the area; and re-aligning the first images to one another the first alignment process again using the images and the one or more additional images to generate an updated first set of the aligned image data. ### Claim 31 (depends on 1) The three-dimensional capture system of claim 1, wherein the pair-wise alignment process comprises identifying overlapping features included in pairs of images and aligning respective images included in the pairs to one another based on the overlapping features. ### Claim 32 (depends on 1) The three-dimensional capture system of claim 1, wherein the global alignment process further comprises determining an alignment between the first set of aligned image data and the second set of aligned imaged data in a graph and optimizing the alignment using a graph optimization process.