# US 2013/0155058 A1 · Four-dimensional augmented reality models for interactive visualization and automated construction progress monitoring > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2013/0155058 A1 - Document ID: US20130155058A1 - Status: Published application - Kind: A1 - Application number: 13/712,362 - Filed: 2012-12-12 - Published: 2013-06-20 - Earliest claimed: 2011-12-14 - Assignees: The Board of Trustees of the University of Illinois - Inventors: Mani Golparvar-Fard; Feniosky A. Peña-Mora; Silvio Savarese - Web page: https://patentgrep.com/patent/US20130155058A1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06Q 10/06311: {Scheduling, planning or task assignment for a person or group} - G06Q 10/103: {Workflow collaboration or project management} - G06T 7/001: {using an image reference approach} - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20076: Probabilistic image processing - G06T 2207/30132: Masonry; Concrete - G06T 2210/04: Architectural design, interior design - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract A method for monitoring construction progress may include storing in memory multiple unordered images obtained from photographs taken at a site; melding the multiple images to reconstruct a dense three-dimensional (3D) as-built point cloud model including merged pixels from the multiple images in 3D space of the site; rectifying and transforming the 3D as-built model to a site coordinate system existing within a 3D as-planned building information model (“as-planned model”); and overlaying the 3D as-built model with the 3D as-planned model for joint visualization thereof to display progress towards completion of a structure shown in the 3D as-planned model. The processor may further link a project schedule to the 3D as-planned model to generate a 4D chronological as-planned model that, when visualized with the 3D as-built point cloud, provides clash detection and schedule quality control during construction. ## Claims ### Claim 1 A computer system comprising: computer storage for storing an unordered group of images obtained from photographs taken at a site; and a processor connected with the storage and configured to meld together the unordered group of images such as to reconstruct dense three-dimensional (3D) and four-dimensional (4D) as-built point cloud models (“as-built models”) comprising merged pixels from the unordered group of images over time, where time is a fourth dimension added to the three dimensions of the site captured from the group of images. ### Claim 2 (depends on 1) The computer system of claim 1, where the group of images comprise low-resolution images that are widely distributed in the site. ### Claim 3 (depends on 1) The computer system of claim 1, where the processor is further configured to rectify and transform the 3D as-built model to a site coordinate system to be jointly visualized with a 3D as-planned building information model (“as-planned model”). ### Claim 4 (depends on 3) The computer system of claim 3, where the processor is further configured to link a project schedule to the 3D as-planned model to generate a 4D chronological as-planned model that, when visualized overlaid with the 4D as-built model, provides clash detection and schedule quality control during building. ### Claim 5 (depends on 4) The computer system of claim 4, where the processor is further configured to register the unordered group of images, which are geo-tagged, with the 4D as-built and as-planned models of the site in an augmented reality environment, where resultant registrations are usable for mutual spatial and temporal comparisons. ### Claim 6 (depends on 4) The computer system of claim 4, where the processor is further configured to track building progress based on an application of the unordered group of images to which is iteratively added on a daily basis, to update the 3D as-built model as compared to the 4D as-planned model. ### Claim 7 (depends on 6) The computer system of claim 6, where the processor is further configured to color code building progress deviations of elements of the 4D as-planned model when compared with elements of the updated 3D as-built model, comprising applying different colors to elements in the 3D as-planned and as-built models according to a status of as-built progress detected within the 4D as-built model. ### Claim 8 (depends on 7) The computer system of claim 7, where the processor is further configured to generate a confidence metric comprising a probability that the color applied to an element to indicate as-built progress is correct, to provide additional data for more precise progress monitoring reports. ### Claim 9 (depends on 6) The computer system of claim 6, where the processor is further configured to apply a probabilistic model that traverses an integrated scene of the site and labels different areas of the integrated scene according to as-built occupancy and as-planned visibility, where the integrated scene comprises an integration of the 4D as-built and as-planned models. ### Claim 10 A computer-implemented method comprising: storing in memory multiple unordered images obtained from photographs taken at a site; melding, with a processor connected with the memory, the multiple images to reconstruct a dense three-dimensional (3D) as-built point cloud model (“as built-model”) comprising merged pixels from the multiple images in 3D space of the site; rectifying and transforming, with the processor, the 3D as-built model to a site coordinate system existing within a 3D as-planned building information model (“as-planned model”); and overlaying, with the processor in a display connected with the processor, the 3D as-built model with the 3D as-planned model for joint visualization thereof to display progress towards completion of a structure shown in the 3D as-planned model. ### Claim 11 (depends on 10) The method of claim 10, further comprising: transforming the 3D as-built model to a four-dimensional (4D) as-built model by updating the 3D as-built model with images from newly taken photographs as they are stored in the memory, and tracking temporal progress of the 3D as-built model as updated. ### Claim 12 (depends on 11) The method of claim 11, further comprising: linking a project schedule to the 3D as-planned model to generate a 4D chronological as-planned model that, when visualized overlaid with the 4D as-built model, provides clash detection and schedule quality control during building; and tracking building progress based on comparing the 4D as-built model to the 4D as-planned model. ### Claim 13 (depends on 12) The method of claim 12, further comprising: registering the multiple images, which are geo-tagged, with the 4D as-built and as-planned models of the site in an augmented reality environment, where resultant registrations are usable for mutual spatial and temporal comparisons. ### Claim 14 (depends on 12) The method of claim 12, further comprising: color coding building progress deviations of elements of the 4D as-planned model when compared with elements of the updated 3D as-built model, comprising applying different colors to elements in the 3D as-planned and as-built models according to a status of as-built progress detected within the 4D as-built model. ### Claim 15 (depends on 14) The method of claim 14, further comprising: generating a confidence metric comprising a probability that the color applied to an element to indicate as-built progress is correct, to provide additional data for more precise progress monitoring reports. ### Claim 16 (depends on 12) The method of claim 12, further comprising: applying a probabilistic model that traverses an integrated scene of the site and labels different areas of the integrated scene according to as-built occupancy and as-planned visibility, where the integrated scene comprises an integration of the 4D as-built and as-planned models. ### Claim 17 A computer-readable storage medium comprising a set of instructions for configuring a construction progress monitoring system executable by a computing device having a processor and memory, the computer-readable medium comprising; instructions to store in the memory multiple unordered images obtained from photographs taken at a site in response to user inputs requesting to store the photographs; instructions to direct the processor to meld the multiple images together from memory to reconstruct a dense three-dimensional (3D) as-built point cloud model (“as built-model”) comprising merged pixels from the multiple images in 3D space of the site; instructions to direct the processor to rectify and transform the 3D as-built model to a site coordinate system existing within a 3D as-planned building information model (“as-planned model”); and instructions to direct the processor to overlay, in a display connected with the processor, the 3D as-built model with the 3D as-planned model for joint visualization thereof to display progress towards completion of a structure shown in the 3D as-planned model. ### Claim 18 (depends on 17) The computer-readable storage medium of claim 17, further comprising a set of instructions to direct the processor to transform the 3D as-built model to a four-dimensional (4D) as-built model by updating the 3D as-built model with images from newly taken photographs as they are stored in the memory, and tracking temporal progress of the 3D as-built model as updated. ### Claim 19 (depends on 18) The computer-readable storage medium of claim 18, further comprising a set of instructions to direct the processor to: link a project schedule to the 3D as-planned model to generate a 4D chronological as-planned model that, when visualized overlaid with the 4D as-built model, provides clash detection and schedule quality control during building; and track building progress based on comparing the 4D as-built model to the 4D as-planned model. ### Claim 20 (depends on 19) The computer-readable storage medium of claim 19, further comprising a set of instructions to direct the processor to register the multiple images, which are geo-tagged, with the 4D as-built and as-planned models of the site in an augmented reality environment, where resultant registrations are usable for mutual spatial and temporal comparisons. ### Claim 21 (depends on 19) The computer-readable storage medium of claim 19, further comprising a set of instructions to direct the processor to: color code building progress deviations of elements of the 4D as-planned model when compared with elements of the updated 3D as-built model, comprising applying different colors to elements in the 3D as-planned and as-built models according to a status of as-built progress detected within the 4D as-built model; and generate a confidence metric comprising a probability that the color applied to an element to indicate as-built progress is correct, to provide additional data for more precise progress monitoring reports. ### Claim 22 (depends on 19) The computer-readable storage medium of claim 19, further comprising a set of instructions to direct the processor to apply a probabilistic model that traverses an integrated scene of the site and labels different areas of the integrated scene according to as-built occupancy and as-planned visibility, where the integrated scene comprises an integration of the 4D as-built and as-planned models.