# US 12,182,960 B1 · Transferring data between 3D and 2D domains

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,960 B1
- Document ID: US12182960B1
- Status: Granted
- Kind: B1
- Application number: 18/642,016
- Filed: 2024-04-22
- Granted: 2024-12-31
- Earliest claimed: 2023-11-03
- Assignees: Dalux Aps
- Applicants: DALUX ApS
- Inventors: Anders Rong
- Web page: https://patentgrep.com/patent/US12182960B1

## CPC

- G06F 30/12: characterised by design entry means specially adapted for CAD, e.g. graphical user interfaces [GUI] specially adapted for CAD
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06F 30/13: Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads
- G06F 30/27: using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/0008: {checking presence/absence}
- G06T 7/11: Region-based segmentation
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/74: Image or video pattern matching; Proximity measures in feature spaces
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 10/817: {by voting}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06F 2111/18: using virtual or augmented reality
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20221: Image fusion; Image merging
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06V 2201/07: Target detection
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for tracking progress of the construction of objects, such as walls of a building, based on 3D digital representation. Building Information Modeling (BIM) may provide a digital representation of the physical and functional characteristics of a place, such as a building comprising walls and other objects. A method includes capturing one or more images comprising pixels at the site with a camera, providing an image location and direction for each image, and for each image location and direction, rendering the building information model into a virtual projection comprising virtual pixels.

## Claims

### Claim 1

A method for monitoring progress of construction of at least one object at a construction site, wherein said object is constructed in multiple stages at said site, said method comprising the steps of:
a. providing a digital representation of a building information model of at least part of said site, wherein said building information model comprises a representation of said at least one object including coordinates of said object with respect to said building information model; and wherein said building information model comprises metadata;
b. capturing one or more images comprising pixels at the site with a camera;
c. providing an image location and direction for each of said images allowing locating and orienting said images with respect to said building information model;
d. for each image location and direction rendering the building information model into a virtual projection comprising virtual pixels; wherein each virtual pixel comprises:
i. the coordinates of the rendered part of the building information model;
ii. a surface normal of the rendered part of the building information model; and
iii. any BIM identification marker of the rendered part of the building information model;
e. using at least one of said images to detect presence of said at least one object or a part of said at least one object applying a machine learning model on said image, wherein the machine learning model has been trained to detect said object or a part of said at least one object;
f. for each image wherein the presence of said object has been detected, identifying the stage of progress of construction of said at least one object applying a machine learning model on said image, wherein the machine learning model has been trained to recognize different stages of said multiple stages;
g. feeding the rendered projection into a digital processor, thereby allowing displaying or comparing a relevant part of the building information model and the at least one image; and
h. presenting the stage of progress of the detected object, optionally with reference to the building information model.

### Claim 2 (depends on 1)

The method according to claim 1, wherein one or both of said steps e. and f. is performed by dividing each of said one or more images into segments, detecting the presence of said object in each segment and/or identifying the stage of progress of any part of said object present in each of said segments, and combining results from the segments to obtain a conclusion concerning the stage of construction of said object.

### Claim 3 (depends on 2)

The method according to claim 2, wherein probabilities are assigned to each of said segments, indicating a likelihood said object is present in each segment, and/or a likelihood of each stage of progress of said object in said segment, followed by making a conclusion of a likely stage of progress of said object based on said assigned probabilities.

### Claim 4 (depends on 1)

The method according to claim 1, wherein one or both of said steps said steps e. and f. is performed by detecting the presence of said object in each of said images and/or identifying the stage of progress of said object present in each of said images without dividing the images into segments.

### Claim 5 (depends on 1)

The method according to claim 1, wherein a final conclusion concerning the stage of construction of said object is obtained by:
iv. performing one or both of said steps said steps e. and f. by dividing each of said one or more images into segments, detecting the presence of said object in each segment and/or identifying the stage of progress of any part of said object present in each of said segments;
v. performing said steps e. and/or f. by detecting the presence of said object in each of said images and/or identifying the stage of progress of said object present in each of said images without dividing the images into segments; and
vi. comparing and combining results from iv. and v. to obtain a final conclusion concerning the stage of construction of said object.

### Claim 6 (depends on 1)

The method according to claim 1, wherein step b. comprises capturing at least two images with a camera from different locations of said site; and the method further comprises steps of localizing the detected objects comprising:
vii. calculating a distance between the different locations;
viii. identifying any common features apparent in said at least two images;
ix. determining the coordinates of any identified common features by triangulation using the coordinates of said locations, the calculated distance between said locations, the direction of capture of said images, and a position of the identified common features in said at least two images;
x. for each identified common feature adding a point to a point cloud representation at the coordinates determined by the triangulation;
xi. aligning the point cloud representation with the building information model; and
xii. using the aligned point cloud representation to confirm the presence of any detected objects with reference to the building information model.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the identification of any common features between said at least two images of step viii. of claim 6 is performed by one or more methods selected among:
xiii. using a machine learning model on individual images; and/or
xiv. using a machine learning model on multiple images.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the identification of common features in different images followed by triangulation to estimate distances are combined with the estimation of distances by using a machine learning model on individual images, to obtain improved distance estimates.

### Claim 9 (depends on 6)

The method according to claim 6, wherein points in the point cloud are stored in an occupancy grid, allowing for distance lookup from any location in the BIM coordinate system.

### Claim 10 (depends on 1)

The method according to claim 1, further comprising:
xv. subdividing a space of the building information model into voxels, wherein said voxels have a length, a height and a depth, and wherein said voxels are smaller than or equal to size of said at least one object;
xvi. using the building information model to determine which of said voxels may contribute at least part of said at least one object, to obtain a set of contributing voxels;
xvii. for each of said contributing voxels of step xvi. determining the stage of progress of construction of the part of said at least one object within said contributing voxel by comparing with the stage of progress of construction results obtained from step f.;
xviii. feeding the stage of progress construction from step xvii. from the set of contributing voxels into a consensus based voting system to identify the stage of progress of construction of said at least one object.

### Claim 11 (depends on 1)

The method according to claim 1, wherein said object is selected among a wall, a panel, an air-vent, and a cable-tray.

### Claim 12 (depends on 1)

The method according to claim 1, wherein said object is a wall, and the stage is selected among not observed, concrete, stud framing, framing with insulation, plasterboard, painted, and finalized.

### Claim 13 (depends on 1)

The method according to claim 1, comprising a step of generating output which is readable on a mobile device, such as a mobile phone or tablet computer, to show the stage of progress of construction of said object.

### Claim 14 (depends on 1)

The method according to claim 1, comprising a step of generating output which is readable on a browser running on a computer to provide a report showing the stage of progress of construction of one or more objects of the building information model.

### Claim 15 (depends on 1)

The method according to claim 1, comprising applying a filter to generate output showing the stage of progress of construction of one or more objects of the building information model, wherein said filter is selectable from one or more categories, preferably the categories are selected among walls, windows, doors, ventilation tubes, ventilation vents, water tubes, electrical installations, and/or fire alert systems.

### Claim 16 (depends on 1)

The method according to claim 1, comprising visually presenting (i) the building information model, and/or (ii) the one or more images, and optionally (iii) any detected objects and the progress of construction of said objects, by one or more of:
I. visual representations of (i) the building information model with a corresponding (ii) image side by side, and optionally (iii) any detected objects and the progress of construction of said objects;
II. a generated composite image of (i) and (ii), and optionally (iii) any detected objects and the progress of construction of said objects, wherein transparency of (i) and/or (ii) may be toggled by a user;
III. the images (ii) and (iii) any detected objects and the progress of construction of said objects;
IV. a two-dimensional floor plan providing information about (iii) any detected objects and the progress of construction of said objects; and
V. a summary report comprising information about and (iii) any detected objects and the progress of construction of said objects.

### Claim 17 (depends on 1)

A system comprising a processor and memory, the system being configured to perform at least steps a, d, e, f, g, and h of claim 1.

### Claim 18

A method for monitoring progress of construction at a construction site, said method comprising the steps of:
a. providing a digital representation of a building information model of at least part of the construction, wherein said building information model comprises metadata including building information model identification markers;
b. capturing one or more images comprising pixels of the construction with a camera;
c. providing an image location and direction for each of said images allowing locating and orienting said images with respect to said building information model;
d. for each image location and direction rendering the building information model into a virtual projection comprising virtual pixels; wherein each virtual pixel comprises:
i. coordinates of the rendered part of the building information model;
ii. a surface normal of the rendered part of the building information model;
iii. any building information model identification marker of the rendered part of the building information model;
iv. optionally, a distance between the image location and the rendered part of the building information model; and
v. optionally, information about whether the rendered part of the building information model faces the camera;
e. feeding the rendered projection into a digital processor, thereby allowing displaying or comparing a relevant part of the building information model and the at least one image; and
f. presenting a stage of progress of the construction, optionally with reference to the building information model.

### Claim 19 (depends on 18)

The method according to claim 18, further comprising, for each image location and direction, using the captured images to calculate and/or estimate for each pixel in the image, at least one of vi., vii., and viii:
vi. the coordinates of the part of the construction, which is visible in the image;
vii. the distance between the camera and the part of the construction, which is visible in the image; and
viii. the surface normal of the part of the construction, which is visible in the image.

### Claim 20 (depends on 19)

The method according to claim 19, wherein said model information of 18, is mapped onto the image information obtained from steps vi., vii., and viii. of claim 19.

### Claim 21 (depends on 20)

The method according to claim 20, wherein the mapped information is used to transfer data from a 3D domain to a 2D domain and/or from the 2D domain to the 3D domain.

### Claim 22 (depends on 18)

The method according to claim 18, wherein step b. comprises capturing at least two images with a camera from different locations of said site; and the method further comprises steps of localizing objects of the building information model comprising:
ix. calculating the distance between the different locations;
x. identifying any common features apparent in said at least two images;
xi. determining the coordinates of any identified common features by triangulation using the coordinates of said locations, the calculated distance between said locations, the direction of capture of said images, and a position of the identified common features in said at least two images;
xii. for each identified common feature adding a point to a point cloud representation at the coordinates determined by the triangulation;
xiii. aligning the point cloud representation with the building information model; and
xiv. using the aligned point cloud representation to confirm a presence of any objects of the building information model.

### Claim 23 (depends on 22)

The method according to claim 22, wherein the identification of any common features between said at least two images of step x. of claim 22 is performed by one or more methods selected among:
a) using a machine learning model on individual images; and/or
b) using a machine learning model on multiple images.

### Claim 24 (depends on 23)

The method according to claim 23, wherein the identification of common features in different images followed by triangulation to estimate distances are combined with the estimation of distances by using a machine learning model on individual images, to obtain improved distance estimates.

### Claim 25 (depends on 22)

The method according to claim 22, wherein points in the point cloud are stored in an occupancy grid, allowing for distance lookup from any location in the building information model coordinate system.

### Claim 26 (depends on 18)

The method according to claim 18, further comprising:
XV. subdividing a space of the building information model into voxels, wherein said voxels have a length, a height and a depth, and wherein said voxels are smaller than or equal to the size of objects of the building information model;
xvi. using the building information model to determine which of said voxels may contribute at least part of objects of the building information model, to obtain a set of contributing voxels;
xvii. for each of said contributing voxels of step xvi. determining the stage of progress of construction of said objects within said contributing voxel;
xviii. feeding the stage of progress of construction of step xvii. from the set of contributing voxels into a consensus based voting system to identify the stage of progress of construction of said objects.

### Claim 27 (depends on 18)

The method according to claim 18, comprising a step of generating output which is readable on a mobile device, such as a mobile phone or tablet computer, to show the stage of progress of construction.

### Claim 28 (depends on 18)

The method according to claim 18, comprising a step of generating output which is readable on a browser running on a computer to provide a report showing the stage of progress of construction of the building information model.

### Claim 29 (depends on 18)

The method according to claim 18, comprising visually presenting (i) the building information model, and/or (ii) the one or more images, and optionally (iii) any detected objects of the building information model and the progress of construction of said objects, by one or more of:
I. visual representations of (i) the building information model with a corresponding (ii) image side by side, and optionally (iii) any detected objects and the progress of construction of said objects;
II. a generated composite image of (i) and (ii), and optionally (iii) any detected objects and the progress of construction of said objects, wherein the transparency of (i) and/or (ii) may be toggled by a user;
III. the images (ii) and (iii) any detected objects and the progress of construction of said objects;
IV. a two-dimensional floor plan providing information about (iii) any detected objects and the progress of construction of said objects; and/or
V. a summary report comprising information about and (iii) any detected objects and the progress of construction of said objects.

### Claim 30 (depends on 18)

A system comprising a processor and memory, the system configured to perform steps a, d, e, and f of claim 18.