# US 12,223,472 B2 · XR-based supervision assisting device, method and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,472 B2
- Document ID: US12223472B2
- Status: Granted
- Kind: B2
- Application number: 17/843,479
- Filed: 2022-06-17
- Published: 2022-12-22
- Granted: 2025-02-11
- Earliest claimed: 2021-06-21
- Assignees: Jae Heon Jung; Yu Mi Lee; SLZ Inc.
- Applicants: Jae Heon Jung; Yu Mi Lee; SLZ Inc.
- Inventors: Jae Heon Jung; Yu Mi Lee
- Web page: https://patentgrep.com/patent/US12223472B2

## CPC

- G06Q 10/20: Administration of product repair or maintenance
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 20/20: in augmented reality scenes
- G06T 2207/30204: Marker
- G06T 2210/04: Architectural design, interior design
- G06T 2219/2021: Shape modification
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure relates to an extended reality (XR)-based supervision assisting device, which finds a construction error by comparing a real design object photographed by a camera of the supervision assisting device with building information modelling (BIM) design data, and corrects the BIM design data to fit the real design object when the construction error between the real design object and the BIM design data exceeds a preset error tolerance range, thereby adjusting the BIM design data according to the construction error.

## Claims

### Claim 1

An extended reality (XR)-based wearable supervision assisting device for assisting supervision of a construction site, the XR-based wearable supervision assisting device comprising:
a camera configured to photograph an image, wherein the camera comprises a gaze tracking camera configured to track a supervisor's gaze based on eye movements of the supervisor;
a transparent display unit;
a communication unit acquiring 3D-based building information modelling (BIM) design data for at least one real design object actually installed in the construction site; and
a processor performing operations of:
grasping the supervisor's gaze through the gaze tracking camera;
determining a real design object, which the supervisor is watching during a predetermined period of time or more, as a recognition target;
acquiring the BIM design data for the determined real design object through the communication unit;
rendering a virtual XR design object for the determined real design object on the basis of the acquired BIM design data;
displaying the rendered XR design object at a position corresponding to the determined real design object on the transparent display unit;
comparing the BIM design data of the XR design object with the photograph image, and checking an error between the XR design object and the determined real design object; and
informing the supervisor of that the checked error deviates from a predetermined allowable error range,
wherein the processor analyzes the BIM design data and derives a supervision progress direction from a starting position of the supervision,
wherein the processor determines a completion of correction of the error in the photographed image when correction of the error of the determined real design object that the supervisor is watching through the transparent display unit has been completed, and
wherein when the completion of correction of the error is determined, the processor renders and displays an XR content guiding a next supervision position on the basis of the supervision progress direction and a current position of the supervision assisting device.

### Claim 2 (depends on 1)

The XR-based wearable supervision assisting device according to claim 1, wherein the processor performs the operations of:
recognizing at least one reference mark in the image;
checking an error by matching the real design object in the image with the BIM design data of the XR design object based on the recognized reference mark; and
specifying a position of the checked error on the basis of current location information of the supervision assisting device and the position of the recognized reference mark.

### Claim 3 (depends on 2)

The XR-based wearable supervision assisting device according to claim 2, wherein the processor performs the operations of:
requesting correction of the BIM design data corresponding to the checked error to a server according to a control signal input to the supervision assisting device; and
re-rendering the virtual XR design object for the real design object on the basis of the corrected BIM design data when receiving the BIM design data in which the error is corrected from the server.

### Claim 4 (depends on 3)

The XR-based wearable supervision assisting device according to claim 3, wherein the server stores location information on at least one reference mark installed in the construction site and the BIM design data, loads the location information of a corresponding reference mark when receiving an image of the reference mark photographed by the camera from the supervision assisting device, and transmits the BIM design data within a predetermined range from the corresponding reference mark to the supervision assisting device.

### Claim 5 (depends on 3)

The XR-based supervision assisting device according to claim 3, wherein when the checked error exceeds an allowable error range, the server corrects a portion of the BIM design data in which the error has occurred to match the real design object, and provides the corrected BIM design data to the supervision assisting device, and
wherein the processor renders a virtual XR design object for the real design object on the basis of the corrected BIM design data received from the server, and provides correction completion information.

### Claim 6 (depends on 3)

The XR-based wearable supervision assisting device according to claim 3, wherein the server generates learning data on the basis of the BIM design data within a predetermined range from the checked error position and performs deep learning by inputting the learning data into an error prediction model,
wherein when a BIM design data of a new supervision object is input, the processor analyzes the BIM design data of the new supervision object based on the error prediction model, predicts an error occurrence position in the new supervision object, and provides the predicted position to the supervision assisting device.

### Claim 7

An extended reality (XR)-based wearable supervision assisting device for assisting supervision of a construction site, the XR-based wearable supervision assisting device comprising:
a camera configured to photograph an image, wherein the camera comprises a gaze tracking camera configured to track a supervisor's gaze based on eye movements of the supervisor;
a display unit;
a communication unit acquiring 3D-based building information modelling (BIM) design data for at least one real design object actually installed in the construction site; and
a processor performing operations of:
grasping the supervisor's gaze through the gaze tracking camera;
determining a real design object, which the supervisor is watching during a predetermined period of time or more, as a recognition target;
acquiring the BIM design data for the determined real design object in the photographed image;
rendering a virtual XR design object for the determined real design object on the basis of the acquired BIM design data;
displaying the rendered XR design object at a position corresponding to the determined real design object on the display unit;
comparing the BIM design data of the XR design object with the photograph image, and checking an error between the XR design object and the determined real design object; and
informing the supervisor of that the checked error deviates from a predetermined allowable error range,
wherein the processor analyzes the BIM design data and derives a supervision progress direction from a starting position of the supervision,
wherein the processor determines a completion of correction of the error in the photographed image when correction of the error of the determined real design object that the supervisor is watching through the display unit has been completed, and
wherein when the completion of correction of the error is determined, the processor renders and displays an XR content guiding a next supervision position on the basis of the supervision progress direction and a current position of the supervision assisting device.

### Claim 8 (depends on 7)

The XR-based wearable supervision assisting device according to claim 7, wherein the processor performs the operations of:
recognizing at least one reference mark in the image;
checking an error by matching the real design object in the image with the BIM design data of the XR design object based on the recognized reference mark; and
specifying a position of the checked error on the basis of current location information of the supervision assisting device and the position of the recognized reference mark.

### Claim 9 (depends on 8)

The XR-based wearable supervision assisting device according to claim 8, wherein the processor performs the operations of:
requesting correction of the BIM design data corresponding to the checked error to a server according to a control signal input to the supervision assisting device; and
re-rendering the virtual XR design object for the real design object on the basis of the corrected BIM design data when receiving the BIM design data in which the error is corrected from the server.

### Claim 10 (depends on 9)

The XR-based wearable supervision assisting device according to claim 9, wherein the server stores location information on at least one reference mark installed in the construction site and the BIM design data, loads the location information of a corresponding reference mark when receiving an image of the reference mark photographed by the camera from the supervision assisting device, and transmits the BIM design data within a predetermined range from the corresponding reference mark to the supervision assisting device.

### Claim 11 (depends on 9)

The XR-based wearable supervision assisting device according to claim 9, wherein when the checked error exceeds an allowable error range, the server corrects a portion of the BIM design data in which the error has occurred to match the real design object, and provides the corrected BIM design data to the supervision assisting device, and
wherein the processor renders a virtual XR design object for the real design object on the basis of the corrected BIM design data received from the server, and provides correction completion information.

### Claim 12 (depends on 9)

The XR-based wearable supervision assisting device according to claim 9, wherein the server generates learning data on the basis of the BIM design data within a predetermined range from the checked error position and performs deep learning by inputting the learning data into an error prediction model,
wherein when a BIM design data of a new supervision object is input, the processor analyzes the BIM design data of the new supervision object based on the error prediction model, predicts an error occurrence position in the new supervision object, and provides the predicted position to the supervision assisting device.

### Claim 13

An extended reality (XR)-based supervision assisting method carried out by an extended reality (XR)-based wearable supervision assisting device for assisting supervision of a construction site, the XR-based wearable supervision assisting method comprising:
grasping a supervisor's gaze through a gaze tracking camera of the XR-based wearable supervision assisting device, based on eye movements of the supervisor;
determining a real design object, which the supervisor is watching through a transparent display unit of the XR-based wearable supervision assisting device during a predetermined period of time or more, as a recognition target;
acquiring building information modelling (BIM) design data for the determined real design object;
rendering a virtual XR design object for the determined real design object on the basis of the acquired BIM design data;
displaying the rendered XR design object at a position corresponding to the determined real design object on the transparent display unit;
comparing the BIM design data of the XR design object with an image photographed by a camera of the XR-based wearable supervision assisting device, and checking an error between the XR design object and the determined real design object;
informing the supervisor of that the checked error deviates from a predetermined allowable error range;
analyzing the BIM design data and derives a supervision progress direction from a starting position of the supervision;
determining a completion of correction of the error in the photographed image when correction of the error of the determined real design object that the supervisor is watching through the transparent display unit has been completed; and
when the completion of correction of the error is determined, rendering and displaying an XR content guiding a next supervision position on the basis of the supervision progress direction and a current position of the supervision assisting device.

### Claim 14 (depends on 13)

The method according to claim 13, wherein the supervision assisting device performs operations of:
recognizing at least one reference mark in the image;
checking an error by matching the real design object in the image with the BIM design data of the XR design object based on the recognized reference mark; and
specifying a position of the checked error on the basis of current location information of the supervision assisting device and the position of the recognized reference mark.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the supervision assisting device performs operations of:
requesting correction of the BIM design data corresponding to the checked error to a server according to a control signal input to the supervision assisting device; and
re-rendering the virtual XR design object for the real design object on the basis of the corrected BIM design data when receiving the BIM design data in which the error is corrected from the server.

### Claim 16 (depends on 15)

The method according to claim 15, wherein the server stores location information on at least one reference mark installed in the construction site and the BIM design data, loads the location information of a corresponding reference mark when receiving an image of the reference mark photographed by the camera from the supervision assisting device, and transmits the BIM design data within a predetermined range from the corresponding reference mark to the supervision assisting device.

### Claim 17 (depends on 15)

The method according to claim 15, wherein when the checked error exceeds an allowable error range, the server corrects a portion of the BIM design data in which the error has occurred to match the real design object, and provides the corrected BIM design data to the supervision assisting device, and
wherein the XR-based wearable supervision assisting device renders a virtual XR design object for the real design object on the basis of the corrected BIM design data received from the server, and provides correction completion information.

### Claim 18 (depends on 13)

A non-transitory computer-readable recording medium which is combined with a computer, which is hardware and stores a program for executing the method of claim 13.