# US 2019/0318548 A1 · Method and apparatus for matching virtual space to real space non-rigid body

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318548 A1
- Document ID: US20190318548A1
- Status: Published application
- Kind: A1
- Application number: 16/383,895
- Filed: 2019-04-15
- Published: 2019-10-17
- Earliest claimed: 2018-04-16
- Assignees: Electronics and Telecommunications Research Institute
- Applicants: Electronics and Telecommunications Research Institute
- Inventors: Yong Sun KIM; Hye Mi KIM; Ki Tae KIM; Ki Hong KIM
- Web page: https://patentgrep.com/patent/US20190318548A1

## CPC

- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 7/11: Region-based segmentation
- G06T 7/251: {involving models}
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present invention relates to a method and an apparatus for matching a virtual object in a virtual environment, the method including generating a point cloud of a non-rigid object, matching a low resolution virtual model to the point cloud, and implementing a high resolution model using the matched model.

## Claims

### Claim 1

A method of matching a virtual space in a virtual environment, the method comprising:
receiving images from one or more cameras;
separating a first object region from the received images and generating a point cloud of the first object; and
matching a virtual mesh model to the generated point cloud,
in which the first object is a non-rigid body object,
the virtual mesh model is a low resolution model, and
a high resolution model of the first object is implemented by using a model generated by matching the virtual mesh model to the point cloud.

### Claim 2 (depends on 1)

The method of claim 1, wherein when the high resolution model is implemented by using the model generated by matching the virtual mesh model to the point cloud,
a vertex of mesh vertices of the generated model is selected as a control point,
a movement of the mesh vertex on the implemented high resolution model corresponding to the control point is calculated, and
the high resolution model is transformed on the basis of the calculated movement of the mesh vertex.

### Claim 3 (depends on 2)

The method of claim 2, wherein a Moving Least Squares algorithm is used.

### Claim 4 (depends on 1)

The method of claim 1, wherein one of the mesh vertices of the implemented high resolution model is selected as a first control point, and
the high resolution model is transformed by tracking the first control point.

### Claim 5 (depends on 4)

The method of claim 4, wherein the matching of the virtual mesh model to the point cloud and the selecting of the first control point, and the transforming of the high resolution model by tracking the first control point are performed in parallel.

### Claim 6 (depends on 4)

The method of claim 4, wherein the first control point is selected by comparing a degree of curvature of the first object implemented with the high resolution model and a color difference between the mesh vertices.

### Claim 7 (depends on 4)

The method of claim 4, wherein when comparing a first control point of a former frame of two consecutive frames of the received image with a first control point of a latter frame, a similarity of vertex colors and a similarity of local 3D structures between the first control points are used as information for tracking the first control point, and
the tracking is performed on the point cloud of the first object.

### Claim 8 (depends on 4)

The method of claim 4, wherein in a case of transforming the high resolution model by tracking the first control point, when the received image is a 2D image,
a second control point is calculated by projecting the selected first control point onto a 2D image coordinate system,
the second control point is tracked,
the first control point is re-acquired by projecting the tracked second control point onto a 3D space, and
the high resolution model is transformed by tracking the acquired first control point.

### Claim 9 (depends on 8)

The method of claim 8, wherein when there are two or more cameras and two or more first control points, one control point is made by performing filtering on the two or more first control points.

### Claim 10 (depends on 8)

The method of claim 8, wherein when comparing a second control point of a former frame of two consecutive frames of the received 2D image with a second control point of a latter frame, information on a similarity of pixel colors and a similarity of patches between the second control points are used as information for tracking the second control points to perform the tracking.

### Claim 11 (depends on 1)

The method of claim 1, wherein the matching is performed by using an Iterative Closest Point algorithm.

### Claim 12

An apparatus for matching a virtual space in a virtual environment, the apparatus comprising:
a matching unit matching a virtual model to a point cloud of a real object, in which the matching unit receives images from one or more cameras;
separates a first object region from the received images and generating a point cloud of the first object; and
matches a virtual mesh model to the generated point cloud,
wherein the first object is a non-rigid body object,
the virtual mesh model is a low resolution model, and
a high resolution model of the first object is implemented by using a model generated by matching the virtual mesh model to the point cloud.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein when the matching unit implements the high resolution model by using the model generated by matching the virtual mesh model to the point cloud,
a vertex of mesh vertices of the generated model is selected as a control point,
a movement of the mesh vertex on the implemented high resolution model corresponding to the control point is calculated, and
the high resolution model is transformed on the basis of the calculated movement of the mesh vertex.

### Claim 14 (depends on 12)

The apparatus of claim 12, wherein the matching unit selects one of the mesh vertices of the implemented high resolution model as a first control point.

### Claim 15 (depends on 14)

The apparatus of claim 14, further comprising a tracking unit, in which the tracking unit transforms the high resolution model by tracking the first control point.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the matching of the virtual mesh model to the point cloud and the selecting of the first control point at the matching unit, and the transforming of the high resolution model by tracking the first control point at the tracking unit are performed in parallel.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein in a case of transforming the high resolution model by tracking the first control point, when the received image is a 2D image,
a second control point is calculated by projecting the selected first control point onto a 2D image coordinate system,
the second control point is tracked,
the first control point is re-acquired by projecting the tracked second control point onto a 3D space, and
the high resolution model is transformed by tracking the acquired first control point.
