# US 12,223,592 B2 · Automatic spatial layout determination and estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,592 B2 - Document ID: US12223592B2 - Status: Granted - Kind: B2 - Application number: 17/559,562 - Filed: 2021-12-22 - Published: 2023-06-22 - Granted: 2025-02-11 - Earliest claimed: 2021-12-22 - Assignees: Streem, LLC - Applicants: Streem, LLC - Inventors: Huapeng Su - Web page: https://patentgrep.com/patent/US12223592B2 ## CPC - G06T 7/579: from motion - G06T 17/05: Geographic models - G01C 11/00: Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/543: from line drawings - G06T 7/62: of area, perimeter, diameter or volume - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 11/00: Two-dimensional [2D] image generation - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/95: {structured as a network, e.g. client-server architectures} - G06V 20/64: Three-dimensional [3D] objects - H04N 23/80: Camera processing pipelines; Components thereof - G01C 11/02: Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures - G06T 2200/08: involving all processing steps from image acquisition to 3D model generation - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2210/04: Architectural design, interior design - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Embodiments include systems and methods for generating 2D and 3D layouts from a physical 3D space captured by a capturing device, the layouts having an identical scale to the physical 3D space, and estimating measurements of the physical 3D space from the layouts. The capturing device captures a point cloud or 3D mesh of the physical 3D space, from which one or more planes are identified. These one or more planes can then be used to create a virtual 3D reconstruction of the captured 3D space. In other embodiments, one plane may be identified as a floor plane, and features from the point cloud or 3D mesh that are above the floor plane may be projected onto the floor plane to create a top down view and 2D layout. Other embodiments are described. ## Claims ### Claim 1 A Method, Comprising: capturing, with a capturing device, a video of a three-dimensional (3D) space by moving or scanning the capturing device about the 3D space; capturing, with the capturing device, motion data of the capturing device as it is moved about the space simultaneously with the captured video; generating, from video and motion data, a 3D mesh representation of the 3D space; assisting a user of the capturing device to complete a closed loop scan for layout estimation of the 3D space by: orienting, using the motion data, the 3D mesh representation into a top-down camera view to simplify current scanning information from 3D to 2D; identifying, from the 3D mesh representation, one or more planes; assigning, from the one or more planes, a plane with a largest area that is located most distant from the top-down camera view as a 2D floor plane; identifying, from the 3D mesh representation, an exterior boundary of the 2D floor plane; identifying, from the 3D mesh representation and the motion data, one or more segments of the exterior boundary located in at least one of the one or more planes; determining, using the motion data, the 3D mesh representation, and the 2D floor plane, a physical layout of each of the one or more planes that is of equal scale to the three-dimensional space, the physical layout including the one or more segments localized onto the at least one of the one or more planes; and displaying the physical layout on a display device to the user to enable the user to adjust a location of at least a portion of the one or more segments or to add and subtract from the one or more segments. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the physical layout of each of the one or more planes comprises determining dimensions of the 2D floor plane, the dimensions reflecting corresponding dimensions of a floor of the three-dimensional space. ### Claim 3 (depends on 1) The method of claim 1, further comprising generating, with the 3D mesh representation and the identified planes, a virtual representation of the three-dimensional space. ### Claim 4 (depends on 1) The method of claim 1, wherein generating the 3D mesh representation of the 3D space comprises calculating a point cloud from the video and motion data. ### Claim 5 (depends on 1) The method of claim 1, wherein the capturing device is a mobile device, including a smartphone or a tablet. ### Claim 6 (depends on 1) The method of claim 1, further comprising generating, from the 3D mesh representation, the 2D floor plan. ### Claim 7 (depends on 6) The method of claim 6, wherein generating the 2D layout comprises projecting, onto the 2D floor plane, one or more protrusions that are above the 2D floor plane. ### Claim 8 A non-transitory computer readable medium (CRM) comprising instructions that, when executed by an apparatus, cause the apparatus to: capture, with a capturing device, depth information of a three-dimensional (3D) space; generate, from the depth information, a top-down 3D representation of the 3D space; assist a user of the capturing device to complete a closed loop scan for layout estimation of the 3D space by: identify, from the 3D representation, a plurality of planes; identify, from the plurality of planes, a floor plane, the floor plane being a plane with a largest area that is located most distant from a top-down camera view of the 3D representation; collapse depth information that is above the floor plane onto the floor plane to obtain a 2D image, where the 2D image reflects the collapsed depth information; detect one or more contours in the 2D image; identify one or more segments located in the one or more detected contours; create, from a contour with a largest area detected in the one or more contours, a 2D layout with a scale that is identical to the 3D space, the 2D layout including the one or more segments localized onto the contour; and display the 2D layout on a display device to the user to enable the user to adjust a location of at least a portion of the one or more segments or to add and subtract from the one or more segments. ### Claim 9 (depends on 8) The CRM of claim 8, wherein the instructions are to further cause the apparatus to smooth the contour with the largest area, and determine, from the smoothed contour, one or more physical dimensions of the three-dimensional space. ### Claim 10 (depends on 9) The CRM of claim 9, wherein the one or more physical dimensions include at least one of a length, an area, or a volume. ### Claim 11 (depends on 8) The CRM of claim 8, wherein the 2D image comprises a plurality of pixels, the depth information comprises a point cloud, and the instructions to cause the apparatus to collapse depth information onto the floor plane are to cause the apparatus to: voxelize each point in the point cloud above the floor plane, and accumulate each voxel above a corresponding pixel in the 2D image by increasing an intensity of the corresponding pixel, the intensity representing a number of voxels above the corresponding pixel. ### Claim 12 (depends on 11) The CRM of claim 11, wherein the instructions to cause the apparatus to detect one or more contours in the 2D image are to cause the apparatus to determine an intensity threshold, and select as a contour all pixels in the 2D image that meet or exceed the intensity threshold. ### Claim 13 (depends on 8) The CRM of claim 8, wherein the instructions to further cause the apparatus to display, on the display device in communication with the apparatus, an interface displaying the 2D layout, and to receive, from an input device in communication with the apparatus, adjustments to the 2D layout. ### Claim 14 (depends on 13) The CRM of claim 13, wherein the display device and the input device comprise a mobile device in communication with the apparatus. ### Claim 15 A system, comprising: a server with a processor, and a storage device in communication with the server, wherein the storage device includes instructions that, when executed by the processor, cause the server to: receive, from a capturing device, a video and correlated depth data of a three-dimensional (3D) space; generate, from video and motion data, a 3D mesh representation of the 3D space; assisting a user of the capturing device to complete a closed loop scan for layout estimation of the 3D space by: orient, using the motion data, the 3D mesh representation into a top-down camera view; identify, from the 3D mesh representation, one or more planes; assign, from the one or more planes, a plane with a largest area that is located most distant from the top-down camera view as a floor plane; identify, from the 3D mesh representation, an exterior boundary of the floor plane; identify, from the 3D mesh representation and motion data, one or more segments of the exterior boundary located in at least one of the one or more planes; determine, from the 3D mesh representation, the floor plane, and one or more identified planes, a 3D physical layout of each of the one or more planes that is of equal scale to the 3D space, the 3D physical layout including the one or more segments localized onto the at least one of the one or more planes; and display the 3D physical layout on a display device to the user to enable the user to adjust a location of at least a portion of the one or more segments of the exterior boundary or to add and subtract from the one or more segments. ### Claim 16 (depends on 15) The system of claim 15, wherein the server is in communication with the capturing device over a network, and the server receives the video and correlated depth data over the network. ### Claim 17 (depends on 16) The system of claim 16, wherein the server is to further transmit, over the network, the 3D physical layout to the capturing device. ### Claim 18 (depends on 17) The system of claim 17, wherein the server is to receive, over the network, designation of a desired measurement of a part of the 3D physical layout, and is to compute results of the desired measurement. ### Claim 19 (depends on 15) The system of claim 15, wherein the instructions are to further cause the server to generate a 2D floor plan from the 3D physical layout.