# US 2023/0267681 A1 · Method of generating three-dimensional model and data processing device performing the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267681 A1
- Document ID: US20230267681A1
- Status: Published application
- Kind: A1
- Application number: 18/153,020
- Filed: 2023-01-11
- Published: 2023-08-24
- Earliest claimed: 2022-02-23
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Changhun SUNG; Byungdeok Kim
- Web page: https://patentgrep.com/patent/US20230267681A1

## CPC

- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 7/32: using correlation-based methods
- G06T 7/33: using feature-based methods
- G06T 7/55: from multiple images
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/90: Determination of colour characteristics
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20072: Graph-based image processing
- G06T 2207/30168: Image quality inspection
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Input images are obtained to include color data and depth data of a target. An image graph is generated based on the color data of the input images such that the image graph includes correlation values between the input images. A registration order is determined based on the image graph. Pose information of each input image is sequentially generated based on the registration order and the depth data of the input images. The input images are sequentially registered as registered images such that each registered image includes the pose information. A three-dimensional model of the target is reconstructed based on the registered images. The pose information may be generated and the elaborate three-dimensional model may be reconstructed by determining the registration order based on the color data and sequentially generating the pose information of the input images according to the registration order using the registered images.

## Claims

### Claim 1

A method of generating a three-dimensional model, comprising:
obtaining, by a data processing device, a plurality of input images such that each input image of the plurality of input images includes color data and depth data of a target;
generating, by the data processing device, an image graph based on the color data of the plurality of input images such that the image graph includes correlation values between the plurality of input images;
determining, by the data processing device, a registration order of the plurality of input images based on the image graph;
sequentially generating, by the data processing device, pose information of each input image with respect to the plurality of input images based on the registration order and the depth data of the plurality of input images;
sequentially registering, by the data processing device, the plurality of input images as registered images such that each registered image includes the pose information; and
reconstructing, by the data processing device, the three-dimensional model of the target based on the registered images.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the image graph includes:
extracting two-dimensional feature points included in each input image based on the color data of each input image;
generating matching information indicating mapping relations of the two-dimensional feature points included in different input images of the plurality of input images; and
determining the correlation values based on the matching information.

### Claim 3 (depends on 2)

The method of claim 2, wherein the determining the correlation values includes:
determining, as the correlation value between two input images among the correlation values, a number of matched pairs of the two-dimensional feature points included in the two input images.

### Claim 4 (depends on 1)

The method of claim 1, wherein the determining the registration order includes:
determining two input images corresponding to a maximum or highest correlation value of the correlation values to be registered firstly.

### Claim 5 (depends on 1)

The method of claim 1, wherein the determining the registration order includes:
determining a sum of the correlation values between each input image that is not registered yet and the input images that are registered already; and
determining the input image corresponding to a maximum or highest value of the sum among the input images that are not registered yet as a present input image to be registered next to the input images that are registered already.

### Claim 6 (depends on 1)

The method of claim 1, wherein the sequentially generating the pose information includes:
generating virtual depth data based on the depth data of the input images that are registered already; and
generating the pose information of a present input image to be registered next to the input images that are registered already based on the virtual depth data and the depth data of the present input image.

### Claim 7 (depends on 6)

The method of claim 6, wherein the generating the virtual depth data includes:
converting the input images that are registered already to three-dimensional images with respect to a same coordinate system; and
generating a depth value per each pixel of the virtual depth data based on an average value of depth values respectively corresponding to the three-dimensional images.

### Claim 8 (depends on 7)

The method of claim 7, wherein the average value of the depth values is obtained by an arithmetic average.

### Claim 9 (depends on 7)

The method of claim 7, wherein the average value of the depth values is obtained by a weighted average using weight values corresponding to the correlation values between the present input image and the input images that are registered already.

### Claim 10 (depends on 6)

The method of claim 6, wherein the generating the pose information of the present input image to be registered next includes:
determining, based on the image graph, a most-correlated input image among the input images that are registered already such that the correlation value between the most-correlated input image and the present input image among the correlation values is greatest among the input images that are registered already; and
determining initial pose information of the present input image based on the depth data of the most-correlated input image and the depth data of the present input image.

### Claim 11 (depends on 10)

The method of claim 10, wherein the generating the pose information of the present input image further includes:
correcting the initial pose information based on the virtual depth data and the depth data of the present input image to generate the pose information of the present input image.

### Claim 12 (depends on 10)

The method of claim 10, wherein the generating the virtual depth data includes:
converting the input images that are registered already to three-dimensional images with respect to a same coordinate system corresponding to the initial pose information; and
generating a depth value per each pixel of the virtual depth data based on an average value of depth values respectively corresponding to the three-dimensional images.

### Claim 13 (depends on 6)

The method of claim 6, wherein the generating the pose information of the present input image includes:
performing an iterative closest point (ICP) algorithm based on the virtual depth data and the depth data of the present input image.

### Claim 14 (depends on 6)

The method of claim 6, wherein the generating the virtual depth data includes:
determining, based on the image graph, non-correlated input images among the input images that are registered already such that the correlation value between the non-correlated input images and the present input image among the correlation values is smaller than a threshold value; and
excluding the non-correlated input images from the input images that are registered already in generating the virtual depth data.

### Claim 15 (depends on 1)

The method of claim 1, further comprising:
determining, based on the image graph, a noisy input image among the plurality of input images such that the correlation values between the noisy input image and other input images are smaller than a threshold value; and
discarding the noisy image.

### Claim 16 (depends on 1)

The method of claim 1, wherein reconstructing the three-dimensional model includes:
generating a three-dimensional point cloud based on the registered images; and
reconstructing the three-dimensional model based on the three-dimensional point cloud.

### Claim 17 (depends on 16)

The method of claim 16, wherein
the generating the three-dimensional point cloud includes improving the three-dimensional point cloud by performing a three-dimensional bundle adjustment with respect to the registered images, and
the reconstructing the three-dimensional model based on the three-dimensional point cloud includes reconstructing a surface of the three-dimensional model using a truncated signed distance function (TSDF).

### Claim 18

A method of generating a three-dimensional model, comprising:
obtaining, by a data processing device, a plurality of input images such that each input image of the plurality of input images includes color data and depth data of a target;
extracting, by the data processing device, two-dimensional feature points included in each input image based on the color data of each input image;
generating, by the data processing device, an image graph based on matching information of the two-dimensional feature points such that the image graph includes correlation values between the plurality of input images;
determining, by the data processing device, a registration order of the plurality of input images based on the image graph;
generating, by the data processing device, virtual depth data based on the depth data of the input images that are registered already;
generating, by the data processing device, pose information of a present input image to be registered next to the input images that are registered already based on the virtual depth data and the depth data of the input images that are registered already;
sequentially registering, by the data processing device, the plurality of input images as registered images such that each registered image includes the pose information; and
reconstructing, by the data processing device, the three-dimensional model of the target based on the registered images.

### Claim 19 (depends on 18)

The method of claim 18, wherein the generating the pose information of the present input image to be registered next includes:
determining, based on the image graph, a most-correlated input image among the input images that are registered already such that the correlation value between the most-correlated input image and the present input image among the correlation values is greatest among the input images that are registered already;
determining an initial pose information of the present input image based on the depth data of the most-correlated input image and the depth data of the present input image; and
correcting the initial pose information based on the virtual depth data and the depth data of the present input image to generate the pose information of the present input image.

### Claim 20

A data processing device comprising:
processing circuitry configured to
receive a plurality of input images such that each input image of the plurality of input images includes color data and depth data of a target,
generate an image graph based on the color data of the plurality of input images such that the image graph includes correlation values between the plurality of input images,
determine a registration order of the plurality of input images based on the image graph,
sequentially generate pose information of each input image with respect to the plurality of input images based on the registration order and the depth data of the plurality of input images,
sequentially register the plurality of input images as registered images such that each registered image includes the pose information; and
reconstruct a three-dimensional model of the target based on the registered images.
