# US 2012/0249807 A1 · Camera and Sensor Augmented Reality Techniques

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0249807 A1
- Document ID: US20120249807A1
- Status: Published application
- Kind: A1
- Application number: 13/078,425
- Filed: 2011-04-01
- Published: 2012-10-04
- Earliest claimed: 2011-04-01
- Assignees: Microsoft Corporation
- Inventors: Benjamin J. Sugden
- Web page: https://patentgrep.com/patent/US20120249807A1

## CPC

- G06T 7/74: {involving reference images or patches}
- G06F 3/147: using display panels
- G06T 7/73: using feature-based methods
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 5/2224: {related to virtual studio applications}
- G06T 2200/04: involving 3D image data
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/30204: Marker
- G06T 2207/30244: Camera pose
- G06T 2210/04: Architectural design, interior design
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Camera and sensor augmented reality techniques are described. In one or more implementations, an optical basis is obtained that was generated from data obtained by a camera of a computing device and a sensor basis is obtained that was generated from data obtained from one or more sensors that are not a camera. The optical basis and the sensor basis describe a likely orientation or position of the camera and the one or more sensors, respectively, in a physical environment. The optical basis and the sensor basis are compared to verify the orientation or the position of the computing device in the physical environment.

## Claims

### Claim 1

A method implemented by a computing device, the method comprising:
obtaining an optical basis generated from data obtained by a camera of the computing device and a sensor basis generated from data obtained from one or more sensors that are not a camera, the optical basis and the sensor basis describing a likely orientation or position of the camera and the one or more sensors, respectively, in a physical environment; and
comparing the optical basis and the sensor basis to verify the orientation or the position of the computing device in the physical environment.

### Claim 2 (depends on 1)

A method as described in claim 1, wherein the one or more sensors include an inertial measurement unit (IMU).

### Claim 3 (depends on 1)

A method as described in claim 1, wherein at least one of the one or more sensors is configured as a gyroscope, an accelerometer, or a magnetometer.

### Claim 4 (depends on 1)

A method as described in claim 1, further comprising calculating a combined basis that indicates the likely orientation or position of the camera and the one or more sensors using the optical basis and the sensor basis.

### Claim 5 (depends on 1)

A method as described in claim 1, further comprising generating an augmentation for display as part of an augmented reality display based on

### Claim 6 (depends on 1)

A method as described in claim 1, further comprising responsive to a determination that the optical basis and the sensor basis do not approximately correspond as a result of the comparing, extrapolating a basis based at least in part on a previous basis of the computing device.

### Claim 7 (depends on 6)

A method as described in claim 6, wherein the extrapolating is based at least in part on the previous basis and the obtained sensor basis.

### Claim 8 (depends on 7)

A method as described in claim 7, wherein the previous basis is computed from a previous optical basis.

### Claim 9 (depends on 1)

A method as described in claim 1, wherein the computing device is configured as a wireless telephone or a tablet.

### Claim 10 (depends on 1)

A method as described in claim 1, wherein the computing device includes a display device that is wearable by a user.

### Claim 11

A method implemented by a computing device, the method comprising:
calculating a combined basis that describes a likely position or orientation of at least a part of the computing device using an optical basis computed from data received from a camera of the computing device and a sensor basis computed from data received from one or more sensors of the computing device; and
generating an augmentation by the computing device for display as part of an augmented-reality display based at least in part on the combined basis.

### Claim 12 (depends on 11)

A method as described in claim 11, wherein the generated augmentation is configured such that an object appears to be added to a physical environment of the computing device.

### Claim 13 (depends on 11)

A method as described in claim 11, wherein the one or more sensors of the computing device include a gyroscope, an accelerometer, or a magnetometer.

### Claim 14 (depends on 11)

A method as described in claim 11, further comprising displaying the augmentation by the computing device using one or more display devices.

### Claim 15

An apparatus comprising:
a camera;
inertial measurement unit (IMU);
a display device; and
one or more modules that are configured to:
compute an optical basis from data obtained by the camera and a sensor basis generated from data obtained from the inertial measurement unit (IMU);
calculate a combined basis that describes a likely orientation or position of the camera and the inertial measurement unit (IMU) in a physical environment; and
cause an augmented-reality display to be displayed by the display device, the augmented-reality display including an augmentation that was generated based at least in part on the combined basis.

### Claim 16 (depends on 15)

An apparatus as described in claim 15, wherein the inertial measure unit (IMU) includes a gyroscope, an accelerometer, and a magnetometer.

### Claim 17 (depends on 15)

An apparatus as described in claim 15, wherein the display device is configured to be wearable by a user.

### Claim 18 (depends on 15)

An apparatus as described in claim 15, wherein the likely orientation or position of the camera and the inertial measure unit (IMU) serve as a basis to determine a like orientation or position of least one body part of a user.

### Claim 19 (depends on 15)

An apparatus as described in claim 15, further comprising a housing that includes the one or more modules and that is configured as a tablet or a wireless phone.

### Claim 20 (depends on 15)

An apparatus as described in claim 15, wherein the augmented-reality display includes the augmentation and an image captured using the camera.
