# US 2024/0005607 A1 · Object scanning using planar segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005607 A1
- Document ID: US20240005607A1
- Status: Published application
- Kind: A1
- Application number: 18/353,592
- Filed: 2023-07-17
- Published: 2024-01-04
- Earliest claimed: 2020-08-20
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Ke-Li CHENG; Kuang-Man HUANG; Michel Adib SARKIS; Gerhard REITMAYR; Ning BI
- Web page: https://patentgrep.com/patent/US20240005607A1

## CPC

- G06T 7/215: Motion-based segmentation
- G06T 17/205: {Re-meshing}
- G06T 7/12: Edge-based segmentation
- G06T 7/55: from multiple images
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Techniques are provided for generating three-dimensional models of objects from one or more images or frames. For example, at least one frame of an object in a scene can be obtained. A portion of the object is positioned on a plane in the at least one frame. The plane can be detected in the at least one frame and, based on the detected plane, the object can be segmented from the plane in the at least one frame. A three-dimensional (3D) model of the object can be generated based on segmenting the object from the plane. A refined mesh can be generated for a portion of the 3D model corresponding to the portion of the object positioned on the plane.

## Claims

### Claim 1

(canceled)

### Claim 2

An apparatus for generating one or more three-dimensional models, comprising:
at least one camera configured to obtain image data of a scene;
at least one depth sensor configured to obtain depth data of a scene;
at least one memory configured to store one or more frames; and
at least one processor coupled to the at least one memory and configured to:
obtain one or more image frames of the scene, the one or more image frames including image data from the at least one camera;
obtain one or more depth frames of the scene, the one or more depth frames including depth data from the at least one depth sensor;
detect a plane within at least one of the one or more image frames of the scene or the one or more depth frames of the scene;
detect an object within at least one of the one or more image frames of the scene or the one or more depth frames of the scene, wherein at least a portion of the object is positioned on the plane;
segment the object from the plane based on at least one of the one or more image frames of the scene or the one or more depth frames of the scene to obtain a segmented object;
generate a partial three-dimensional (3D) model of the segmented object based on at least one of the one or more image frames or the one or more depth frames; and
generate a refined 3D model based on the partial 3D model and at least the one or more depth frames of the scene, the refined 3D model including the portion of the object positioned on the plane.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the at least one processor is configured to:
render the refined 3D model for display; and
receive user input associated with manipulate the refined 3D model.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the at least one processor is configured to:
modify the refined 3D model based on the user input to generate a modified 3D model; and
render the modified 3D model.

### Claim 5 (depends on 2)

The apparatus of claim 2, wherein the at least one processor is configured to:
apply texture to the refined 3D model.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the at least one processor is configured to:
apply the texture to a portion of the refined 3D model corresponding to at least the portion of the object positioned on the plane.

### Claim 7 (depends on 2)

The apparatus of claim 2, wherein the at least one processor is configured to:
generate the partial 3D model of the segmented object based on a 3D point cloud of the object.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein the at least one processor is configured to:
generate a bounding box based on the 3D point cloud.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein, to generate the refined 3D model, the at least one processor is configured to:
remove points from the 3D point cloud that are outside of the bounding box.

### Claim 10 (depends on 2)

The apparatus of claim 2, wherein the at least one processor is configured to:
track the plane from a first frame of the one or more image frames to a second frame of the one or more image frames;
segment, based on tracking the plane, the object from the plane in the second frame; and
generate the partial 3D model of the segmented object based on segmenting the object from the plane in the second frame.

### Claim 11 (depends on 2)

The apparatus of claim 2, wherein the at least one processor is configured to:
output the refined 3D model for display.

### Claim 12 (depends on 11)

The apparatus of claim 11, further comprising a display configured to display the refined 3D model.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the apparatus is an extended reality device.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein a real-world environment is viewable through a display of the extended reality device as the refined 3D model is displayed by the display.

### Claim 15 (depends on 2)

The apparatus of claim 2, wherein the at least one processor is configured to:
determine at least one keyframe from the one or more image frames of the scene; and
generate the partial 3D model based on the at least one keyframe.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the at least one processor is configured to determine the at least one keyframe based on an amount of overlap between at least two image frames of the one or more image frames of the scene.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein, to determine the at least one keyframe, the at least one processor is configured to:
determine an amount of overlap between a first frame of the one or more image frames and a second frame of the one or more frames is less than an overlap threshold; and
determine the first frame as a keyframe based on the amount of overlap between the first frame and the second frame being less than an overlap threshold.

### Claim 18

A method of generating one or more three-dimensional models, the method comprising:
obtaining one or more image frames of a scene, the one or more image frames including image data from at least one camera;
obtaining one or more depth frames of the scene, the one or more depth frames including depth data from at least one depth sensor;
detecting a plane within at least one of the one or more image frames of the scene or the one or more depth frames of the scene;
detecting an object within at least one of the one or more image frames of the scene or the one or more depth frames of the scene, wherein at least a portion of the object is positioned on the plane;
segmenting the object from the plane based on at least one of the one or more image frames of the scene or the one or more depth frames of the scene to obtain a segmented object;
generating a partial three-dimensional (3D) model of the segmented object based on at least one of the one or more image frames or the one or more depth frames; and
generating a refined 3D model based on the partial 3D model and at least the one or more depth frames of the scene, the refined 3D model including the portion of the object positioned on the plane.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
rendering the refined 3D model for display; and
receiving user input associated with manipulate the refined 3D model.

### Claim 20 (depends on 19)

The method of claim 19, further comprising:
modifying the refined 3D model based on the user input to generate a modified 3D model; and
rendering the modified 3D model.

### Claim 21 (depends on 18)

The method of claim 18, further comprising:
applying texture to the refined 3D model.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
applying the texture to a portion of the refined 3D model corresponding to at least the portion of the object positioned on the plane.

### Claim 23 (depends on 18)

The method of claim 18, further comprising:
generating the partial 3D model of the segmented object based on a 3D point cloud of the object.

### Claim 24 (depends on 23)

The method of claim 23, further comprising:
generating a bounding box based on the 3D point cloud.

### Claim 25 (depends on 24)

The method of claim 24, wherein generating the refined 3D model comprises:
removing points from the 3D point cloud that are outside of the bounding box.

### Claim 26 (depends on 18)

The method of claim 18, further comprising:
tracking the plane from a first frame of the one or more image frames to a second frame of the one or more image frames;
segmenting, based on tracking the plane, the object from the plane in the second frame; and
generating the partial 3D model of the segmented object based on segmenting the object from the plane in the second frame.

### Claim 27 (depends on 18)

The method of claim 18, wherein a real-world environment is viewable through a display of an extended reality device as the refined 3D model is displayed by a display.

### Claim 28 (depends on 18)

The method of claim 18, further comprising:
determining at least one keyframe from the one or more image frames of the scene; and
generating the partial 3D model based on the at least one keyframe.

### Claim 29 (depends on 28)

The method of claim 28, wherein the at least one keyframe is determined based on an amount of overlap between at least two image frames of the one or more image frames of the scene.

### Claim 30 (depends on 28)

The method of claim 28, wherein determining the at least one keyframe comprises:
determining an amount of overlap between a first frame of the one or more image frames and a second frame of the one or more frames is less than an overlap threshold; and
determining the first frame as a keyframe based on the amount of overlap between the first frame and the second frame being less than an overlap threshold.
