# US 12,002,162 B2 · Method and apparatus for providing virtual contents in virtual space based on common coordinate system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,162 B2 - Document ID: US12002162B2 - Status: Granted - Kind: B2 - Application number: 17/607,402 - Filed: 2020-07-27 - Published: 2022-06-30 - Granted: 2024-06-04 - Earliest claimed: 2019-08-14 - Assignees: Korea Institute of Science and Technology - Applicants: Korea Institute of Science and Technology - Inventors: Byoung Hyun Yoo; Yong Jae Lee - Web page: https://patentgrep.com/patent/US12002162B2 ## CPC - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/003: {Navigation within 3D models or images} - G06T 7/11: Region-based segmentation - G06T 7/73: using feature-based methods - G06T 7/74: {involving reference images or patches} - G06V 10/225: {based on a marking or identifier characterising the area} - G06V 10/24: Aligning, centring, orientation detection or correction of the image - G06V 10/7553: {based on shape, e.g. active shape models [ASM]} - G06V 20/653: {by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces} - G06T 2207/30204: Marker - G06T 2207/30208: Marker matrix - G06T 2219/024: Multi-user, collaborative environment - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for providing virtual contents in a virtual space based on a common coordinate system includes: detecting a base marker for identifying a fixed point of an actual operation space, and a target marker for identifying an actual operation object, from initial image data indicating an initial state of the actual operation object in the actual operation space; calculating an initial model matrix of a target marker expressing an initial position and orientation of the target marker, and a model matrix of the base marker expressing a position and an orientation of the detected base marker in a virtual operation space having a common coordinate system which has the detected base marker as a reference; and calculating a current model matrix of the target marker expressing a current position and orientation of the target marker by using the calculated model matrix of the base marker. ## Claims ### Claim 1 A method for providing virtual contents in a virtual space based on a common coordinate system, the method comprising: a step of detecting a base marker, which is a marker for identifying a fixed point of an actual operation space, and a target marker, which is a marker for identifying an actual operation object, from initial image data indicating an initial state of the actual operation object in the actual operation space; a step of calculating an initial model matrix of the target marker expressing an initial position and orientation of the target marker, and a model matrix of the base marker expressing a position and an orientation of the detected base marker in a virtual operation space having a common coordinate system which uses the detected base marker as a reference; a step of calculating a current model matrix of the target marker expressing a current position and orientation of the target marker by using the calculated model matrix of the base marker; a step of generating a virtual operation object within the virtual operation space, the virtual operation object comprising a three-dimensional model shape of the actual operation object; and a step of mirroring changes in a position and an orientation of the actual operation object onto the virtual operation object, said step comprising: detecting, via a camera module of a field device in the actual operation space, a difference between the current model matrix of the target marker and the initial model matrix of the target marker; and updating at least one of a position and an orientation of the virtual operation object according to the difference. ### Claim 2 (depends on 1) The method for providing virtual contents in a virtual space based on a common coordinate system according to claim 1, wherein in the step of detecting the target marker, a shape included in the initial image data is extracted, and the base marker and the target marker are detected by using the extracted shape. ### Claim 3 (depends on 2) The method for providing virtual contents in a virtual space based on a common coordinate system according to claim 2, wherein in the step of detecting the target marker, the extracted shape is compared with a shape of the base marker and a shape of the target marker, a shape matching the shape of the base marker is detected as the base marker, and a shape matching the shape of the target marker is detected as the target marker. ### Claim 4 (depends on 1) The method for providing virtual contents in a virtual space based on a common coordinate system according to claim 1, wherein the step of calculating the model matrix of the base marker includes: a step of calculating an initial model view matrix of the base marker indicating the position and the orientation of the base marker using as a reference a view of a camera module used for photographing of the initial image data based on the initial image data, and an initial model view matrix of the target marker indicating the position and the orientation of the target marker using as a reference the view of the camera module; a step of calculating the initial model matrix of the target marker on the virtual operation space; and a step of calculating a model matrix of the base marker by using the calculated initial model view matrix of the base marker, the calculated initial model matrix of the target marker, and the calculated initial model view matrix of the target marker. ### Claim 5 (depends on 4) The method for providing virtual contents in a virtual space based on a common coordinate system according to claim 4, wherein in the step of calculating the model matrix of the base marker by using the calculated initial model view matrix of the base marker, the calculated initial model matrix of the target marker, and the calculated initial model view matrix of the target marker, the model matrix of the base marker is calculated according to the following equation, Mb matrix=Mo matrix*(MVo matrix)−1*MVb matrix (Mb matrix: model matrix of base marker, MVb matrix: initial model view matrix of base marker, Mo matrix: initial model matrix of target marker, MVo matrix: initial model view matrix of target marker). ### Claim 6 (depends on 1) The method for providing virtual contents in a virtual space based on a common coordinate system according to claim 1, wherein the step of calculating the current model matrix of the target marker includes: a step of calculating a current model view matrix of the base marker and a current model view matrix of the target marker based on the current image data; and a step of calculating a current model matrix of the target marker by using the calculated current model view matrix of the base marker, the calculated current model view matrix of the target marker, and the calculated model matrix of the base marker. ### Claim 7 (depends on 6) The method for providing virtual contents in a virtual space based on a common coordinate system according to claim 6, wherein in the step of calculating the current model matrix of the target marker by using the calculated current model view matrix of the base marker, the calculated current model view matrix of the target marker, and the calculated model matrix of the base marker, the current model matrix of the target marker is calculated according to the following equation, Mo′matrix=Mb matrix*(MVb′matrix)−1*MVo′matrix (Mo′ matrix: current model matrix of target marker, MVo′ matrix: current model view matrix of target marker, MVb′ matrix: current model view matrix of base marker, Mb matrix: model matrix of base marker). ### Claim 8 (depends on 1) The method for providing virtual contents in a virtual space based on a common coordinate system according to claim 1, further comprising: a step of determining whether a command for changing a position and an orientation of the virtual operation object is received from a remote device of a remote operator; and a step of mirroring changes in the position and the orientation of the virtual operation object on the virtual operation space according to a received command when a command for changing the position and the orientation of the virtual operation object is received. ### Claim 9 (depends on 1) The method for providing virtual contents in a virtual space based on a common coordinate system according to claim 1, further comprising: a step of updating the position and the orientation of the virtual operation object on the virtual operation space according to the calculated current model matrix of the target marker. ### Claim 10 (depends on 1) A computer-readable recording medium in which a program for executing the method according to claim 1 in a computer is recorded. ### Claim 11 A device comprising: a marker detection unit that detects a base marker, which is a marker for identifying a fixed point of an actual operation space, and a target marker, which is a marker for identifying an actual manipulation object, from initial image data indicating an initial state of the actual manipulation object in the actual operation space; a matrix calculation unit which calculates an initial model matrix of the target marker expressing an initial position and orientation of the target marker, and a model matrix of the base marker expressing a position and an orientation of the detected base marker in a virtual operation space having a common coordinate system which uses the detected base marker as a reference, and calculates a current model matrix of the target marker expressing a current position and orientation of the target marker by using the calculated model matrix of the base marker; and a virtual space processing unit that generates a virtual operation object within the virtual operation space, the virtual operation object comprising a three-dimensional model shape of the actual operation object, and mirrors changes in a position and an orientation of the actual operation object onto the virtual operation object; wherein the virtual space processing unit is configured to mirror changes in the position and the orientation of the actual operation object onto the virtual operation object by: detecting, via a camera module of a field device in the actual operation space, a difference between the current model matrix of the target marker and the initial model matrix of the target marker; and updating at least one of a position and an orientation of the virtual operation object according to the difference.