# US 2012/0075342 A1 · Augmenting image data based on related 3d point cloud data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0075342 A1
- Document ID: US20120075342A1
- Status: Published application
- Kind: A1
- Application number: 12/890,251
- Filed: 2010-09-24
- Published: 2012-03-29
- Earliest claimed: 2010-09-24
- Assignees: Intel Corporation
- Inventors: Maha El Choubassi; Igor V. Kozintsev; Yi Wu; Yoram Gat; Horst Haussecker
- Web page: https://patentgrep.com/patent/US20120075342A1

## CPC

- G06T 7/11: Region-based segmentation
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Embodiments of the invention describe processing a first image data and 3D point cloud data to extract a first planar segment from the 3D point cloud data. This first planar segment is associated with an object included in the first image data. A second image data is received, the second image data including the object captured in the first image data. A second planar segment related to the object is generated, where the second planar segment is geometrically consistent with the object as captured in the second image data. This planar segment is generated based, at least in part, on the second image data, the first image data and the first planar segment. Embodiments of the invention may further augment the second image data with content associated with the object. This augmented image may be displayed such that the content is displayed geometrically consistent with the second planar segment.

## Claims

### Claim 1

An article of manufacture comprising a non-transitory machine-readable storage medium that provides instructions that, if executed by the machine, will cause the machine to perform operations comprising:
processing a first image data and 3D point cloud data to extract a first planar segment from the 3D point cloud data, the first planar segment associated with an object included in the first image data;
generating a second planar segment related to the object as captured in a second image data based, at least in part, on the second image data, the first image data and the first planar segment, the second image data captured via an image sensor included in a mobile computing device, the second image data having the object; and
augmenting, geometrically consistent with the second planar segment, the second image data with content associated with the object.

### Claim 2 (depends on 1)

The article of manufacture of claim 1, wherein processing the first image data and the 3D point cloud data to extract the first planar segment includes determining the shape and orientation of the first extracted planar segment based on metadata associated with the first image data, the metadata having geographical information of the first image data or data related to the object captured in the first image data.

### Claim 3 (depends on 1)

The article of manufacture of claim 1, wherein generating the second planar segment related to the object as captured in the second image data is based, at least in part, on the second image data, the first image data and the first planar segment includes matching the first planar segment with the object captured in the second image data based on color data of the object captured in the first image data.

### Claim 4 (depends on 1)

The article of manufacture of claim 1 wherein processing the 3D point cloud data includes downsampling the 3D point cloud data.

### Claim 5 (depends on 1)

The article of manufacture of claim 1, wherein processing the first image data and the 3D point cloud data to extract the first planar segment from the 3D point cloud data is further based on reliability data associated with data points of the 3D point cloud data.

### Claim 6 (depends on 1)

The article of manufacture of claim 1, wherein the second image data comprises a live view captured by the image sensor of the mobile computing device, and the content associated with the object includes text, static image data or video image data.

### Claim 7 (depends on 1)

The article of manufacture of claim 1, wherein the 3D point cloud data includes
a first set of 3D point cloud data captured from a first viewpoint; and
a second set of 3D point cloud data captured from a second viewpoint.

### Claim 8 (depends on 7)

The article of manufacture of claim 7, wherein processing the first image data and the 3D point cloud data to extract the first planar segment from the 3D point cloud data includes
processing the first set of 3D point cloud data and the first image data to extract a first initial planar segment;
processing the second set of 3D point cloud data and the first image data to extract a second initial planar segment; and
combining the first and second initial planar segments to form the first planar segment.

### Claim 9

A system comprising:
at least one processor;
a display;
software to be executed via the processor(s) to
process a first image data and 3D point cloud data to extract a first planar segment from the 3D point cloud data, the first planar segment associated with an object included in the first image data,
generate a second planar segment related to the object as captured in a second image data based, at least in part, on the second image data, the first image data and the first planar segment,
augment, geometrically consistent with the second planar segment, the second image data with content associated with the object,
display the second image with the augmented content associated with the object on the display; and
an image sensor to capture the second image data having the object.

### Claim 10 (depends on 9)

The system of claim 9, wherein processing the first image data and the 3D point cloud data to extract the first planar segment includes determining the shape and orientation of the first extracted planar segment based on metadata associated with the first image data, the metadata to include geographical information of the first image data or data related to the object captured in the first image data.

### Claim 11 (depends on 9)

The system of claim 9, wherein generating the second planar segment related to the object as captured in the second image data is based, at least in part, on the second image data, the first image data and the first planar segment includes matching the first planar segment with the object captured in the second image data based on color data of the object captured in the first image data.

### Claim 12 (depends on 9)

The system of claim 9 wherein processing the 3D point cloud data includes downsampling the 3D point cloud data.

### Claim 13 (depends on 9)

The system of claim 9, wherein processing the first image data and the 3D point cloud data to extract the first planar segment from the 3D point cloud data is further based on reliability data associated with data points of the 3D point cloud data.

### Claim 14 (depends on 9)

The system of claim 9, wherein the second image data comprises a live view captured by the image sensor of the mobile computing device, and the content includes text, static image data or video image data.

### Claim 15 (depends on 9)

The system of claim 9, wherein the 3D point cloud data includes
a first set of 3D point cloud data captured from a first viewpoint; and
a second set of 3D point cloud data captured from a second viewpoint.

### Claim 16 (depends on 15)

The system of claim 15, wherein processing the first image data and the 3D point cloud data to extract the first planar segment from the 3D point cloud data includes
processing the first set of 3D point cloud data and the first image data to extract a first initial planar segment;
processing the second set of 3D point cloud data and the first image data to extract a second initial planar segment; and
combining the first and second initial planar segments to form the first planar segment.

### Claim 17

A method comprising:
processing a first image data and 3D point cloud data to extract a first planar segment from the 3D point cloud data, the first planar segment associated with an object included in the first image data;
receiving a second image data captured via an image sensor included in a mobile computing device, the second image data to include the object;
generating a second planar segment related to the object as captured in the second image data based, at least in part, on the second image data, the first image data and the first planar segment; and
augmenting, geometrically consistent with the second planar segment, the second image data with content associated with the object.

### Claim 18 (depends on 17)

The method of claim 17, wherein processing the first image data and the 3D point cloud data to extract the first planar segment includes determining the shape and orientation of the first extracted planar segment based on metadata associated with the first image data, the metadata to include geographical information of the first image data or data related to the object captured in the first image data.

### Claim 19 (depends on 17)

The method of claim 17, wherein generating the second planar segment related to the object as captured in the second image data is based, at least in part, on the second image data, the first image data and the first planar segment includes matching the first planar segment with the object captured in the second image data based on color data of the object captured in the first image data.

### Claim 20 (depends on 17)

The method of claim 17, wherein processing the 3D point cloud data includes downsampling the 3D point cloud data.

### Claim 21 (depends on 17)

The method of claim 17, wherein processing the first image data and the 3D point cloud data to extract the first planar segment from the 3D point cloud data is further based on reliability data associated with data points of the 3D point cloud data.

### Claim 22 (depends on 17)

The method of claim 17, wherein the second image data comprises a live view captured by the image sensor of the mobile computing device, and the content includes text, static image data or video image data.

### Claim 23 (depends on 17)

The method of claim 17, wherein the 3D point cloud data includes
a first set of 3D point cloud data captured from a first viewpoint; and
a second set of 3D point cloud data captured from a second viewpoint.

### Claim 24 (depends on 23)

The method of claim 23, wherein processing the first image data and the 3D point cloud data to extract the first planar segment from the 3D point cloud data includes
processing the first set of 3D point cloud data and the first image data to extract a first initial planar segment;
processing the second set of 3D point cloud data and the first image data to extract a second initial planar segment; and
combining the first and second initial planar segments to form the first planar segment.
