# US 9,519,973 B2 · Enabling use of three-dimensional locations of features images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,519,973 B2
- Document ID: US9519973B2
- Status: Granted
- Kind: B2
- Application number: 14/020,873
- Filed: 2013-09-08
- Published: 2015-03-12
- Granted: 2016-12-13
- Earliest claimed: 2013-09-08
- Assignees: Intel Corporation
- Applicants: RealityCap, Inc.
- Inventors: Eagle Sunrise Jones; Benjamin J. Hirashima; Jordan H. Miller
- Web page: https://patentgrep.com/patent/US9519973B2

## CPC

- G06T 7/75: {involving models}
- G06T 7/579: from motion
- G06T 7/74: {involving reference images or patches}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The disclosed embodiments provide a system that facilitates use of an image. During operation, the system uses a set of images from a camera on a device to obtain a set of features in proximity to the device, wherein the set of images comprises the image. Next, the system uses the set of images and inertial data from one or more inertial sensors on the device to obtain a set of three-dimensional (3D) locations of the features. Finally, the system enables use of the set of 3D locations with the image.

## Claims

### Claim 1

A computer-implemented method for facilitating use of an image, comprising:
using a set of images from a camera on a device to obtain a set of features in proximity to the device;
using the set of images and inertial data from one or more inertial sensors on the device to generate a three-dimensional (3D) model of an environment around the device;
obtaining a selection of the image from the set of images;
selecting, as a subset of the generated 3D model, a set of 3D locations of the features and a set of 2D locations of the features in the image;
storing the selected subset of the 3D model comprising the 3D locations and the 2D locations as metadata for the image for subsequent use of the 3D locations with the features in the image; and
omitting a remainder of the 3D model that is not in the subset from the metadata.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
enabling use of the 3D locations with one or more additional images from the set of images.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the image and the one or more additional images are associated with at least one of:
a panorama; and
two or more images from an image sequence.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein using the set of images and the inertial data from the one or more inertial sensors to generate the 3D model of the environment around the device comprises:
obtaining a new image from the set of images and a measurement of a 2D location, in the new image, of a feature from the set of features;
providing the inertial data and the measurement as input to generate the 3D model;
using the 3D model to predict a 2D location of the feature in the new image;
and
using a residual between the predicted 2D location and the measurement to estimate a 3D location of the feature.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the 3D locations and the 2D locations are embedded in the image.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
obtaining a selection of one or more points in the image; and
using the 3D locations of one or more features in the metadata to measure one or more attributes associated with the one or more points.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the one or more attributes comprise at least one of:
a distance;
an area;
a volume; and
a dimension.

### Claim 8 (depends on 6)

The method of claim 6, wherein the one or more points comprise at least one of:
a feature from the set of features; and
a point in a shape defined using the feature.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the set of features comprises at least one of:
a corner;
an edge; and
a specialized feature.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the device is a portable electronic device.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
displaying graphical objects identifying the 2D locations of the features in the image.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
using the metadata to calculate a 3D location of a point in the image that is not included in the 2D locations of the features.

### Claim 13

A system for facilitating use of an image, comprising:
a camera configured to obtain a set of images;
one or more inertial sensors configured to obtain inertial data associated with the camera;
an analysis apparatus configured to:
use the set of images to obtain a set of features in proximity to the camera;
use the set of images and the inertial data to generate a three-dimensional (3D) model of an environment around the device;
obtain a selection of the image from the set of images; and
select, as a subset of the generated 3D model, a set of 3D locations of the features and a set of 2D locations of the features in the image; and
a management apparatus configured to:
store the selected subset of the 3D model comprising the 3D locations and the 2D locations as metadata for the image for subsequent use of the 3D locations with the features in the image; and
omit a remainder of the 3D model that is not in the subset from the metadata.

### Claim 14 (depends on 13)

The system of claim 13, wherein using the set of images and the inertial data to generate the 3D model of the environment around the device comprises:
obtaining a new image from the set of images and a measurement of a 2D location, in the new image, of a feature from the set of features;
providing the inertial data and the measurement as input to generate the 3D model;
using the 3D model to predict a 2D location of a feature from the set of features in the new image;
and
using a residual between the predicted 2D location and the measurement to estimate a 3D location of the feature.

### Claim 15 (depends on 13)

The system of claim 13, wherein the management apparatus is further configured to:
obtain a selection of one or more points in the image; and
use the 3D locations of one or more features in the metadata to measure one or more attributes associated with the one or more points.

### Claim 16 (depends on 15)

The system of claim 15, wherein the one or more attributes comprise at least one of:
a distance;
an area;
a volume; and
a dimension.

### Claim 17

A non-transitory computer-readable storage medium containing instructions embodied therein for causing a computer system to perform a method for facilitating use of an image, the method comprising:
using a set of images from a camera on a device to obtain a set of features in proximity to the device;
using the set of images and inertial data from one or more inertial sensors on the device to generate a three-dimensional (3D) model of an environment around the device;
obtaining a selection of the image from the set of images;
selecting, as a subset of the generated 3D model, a set of 3D locations of the features and a set of 2D locations of the features in the image;
storing the selected subset of the 3D model comprising the 3D locations and the 2D locations as metadata for the image for subsequent use of the 3D locations with the features in the image; and
omitting a remainder of the 3D model that is not in the subset from the metadata.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, the method further comprising:
enabling use of the 3D locations with one or more additional images from the set of images.

### Claim 19 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, the method further comprising:
obtaining a selection of one or more points in the image; and
using the 3D locations of one or more features in the metadata to measure one or more attributes associated with the one or more points.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium of claim 19, wherein the one or more attributes comprise at least one of:
a distance;
an area;
a volume; and
a dimension.