# US 2015/0103183 A1 · Method and apparatus for device orientation tracking using a visual gyroscope

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0103183 A1
- Document ID: US20150103183A1
- Status: Published application
- Kind: A1
- Application number: 14/051,282
- Filed: 2013-10-10
- Published: 2015-04-16
- Earliest claimed: 2013-10-10
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Josh Abbott; Jared Heinly; Jim van Welzen
- Web page: https://patentgrep.com/patent/US20150103183A1

## CPC

- H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability}
- G01C 21/1656: {with passive imaging devices, e.g. cameras}
- G06T 7/248: {involving reference images or patches}
- G06T 7/74: {involving reference images or patches}
- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- H04N 23/63: by using electronic viewfinders
- H04N 23/682: {Vibration or motion blur correction}
- G01C 11/00: Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30244: Camera pose
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for tracking device orientation on a portable device is disclosed. The method comprises initializing a device orientation to a sensor orientation, wherein the sensor orientation is based on information from an inertial measurement unit (IMU) sensor. It also comprises initiating visual tracking using a camera on the portable device and capturing a frame. Next, it comprises determining a plurality of visual features in the frame and matching the frame to a keyframe, wherein capture of the keyframe precedes capture of the frame. Subsequently, it comprises computing a rotation amount between the frame and the keyframe. Responsive to a determination that a rotational distance between the frame and the keyframe exceeds a predetermined threshold, promoting the frame to a keyframe status and adding it to a first orientation map and adjusting the frame with all prior captured keyframes.

## Claims

### Claim 1

A method for tracking device orientation on a portable device, said method comprising:
initializing a device orientation to a sensor orientation, wherein said sensor orientation is based on information from an inertial measurement unit (IMU) sensor;
initiating visual tracking using a camera on said portable device;
capturing a frame using said camera;
determining a plurality of visual features in said frame;
matching said frame to a keyframe, wherein capture of said keyframe precedes capture of said frame, and wherein said keyframe was captured using visual tracking;
computing a rotation amount between said frame and said keyframe;
responsive to a determination that a rotational distance between said frame and said keyframe exceeds a predetermined threshold, promoting said frame to a keyframe status and adding said frame to a first orientation map; and
performing an adjustment of said frame with all prior captured keyframes.

### Claim 2 (depends on 1)

The method of claim 1, wherein said determining comprises:
computing repeatable and distinct features in said frame using a feature detecting procedure; and
detecting and describing said features using a feature description procedure.

### Claim 3 (depends on 2)

The method of claim 2, wherein said feature detecting procedure is selected from a group consisting of: FAST, Harris & Stephens, Plessey, Shi-Tomasi, Moravec corner detection, Wang and Brady corner detection, and SUSAN corner detector.

### Claim 4 (depends on 2)

The method of claim 2, wherein said feature description procedure is selected from a group consisting of: SURF, SIFT and BRIEF.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
responsive to a determination that a rotational distance between said frame and said keyframe is within said predetermined threshold, continuing to search for next keyframe.

### Claim 6 (depends on 1)

The method of claim 1, wherein said matching is performed using RANSAC.

### Claim 7 (depends on 1)

The method of claim 1, wherein said computing is performed using Horn's procedure.

### Claim 8 (depends on 1)

The method of claim 1 further comprising:
responsive to a loss of said visual tracking, determining device orientation by combining delta values from said IMU sensor with a last known orientation measurement from said visual tracking; and
building a second orientation map, wherein said second orientation map is created using data from said IMU sensor.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
determining if said second orientation map overlaps with keyframes from said orientation map; and
responsive to a determination of overlap, deleting said second orientation map and continuing to build said first orientation map.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
determining if said second orientation map overlaps with keyframes from said first orientation map; and
responsive to a determination of overlap, merging said second orientation map with said first orientation map.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
determining if said second orientation map overlaps with keyframes from said first orientation map; and
responsive to a determination of no overlap, continuing to build said second orientation map.

### Claim 12

A computer-readable storage medium having stored thereon instructions that, if executed by a computer system cause the computer system to perform a method for tracking device orientation on a portable device, said method comprising:
initializing a device orientation to a sensor orientation, wherein said sensor orientation is based on information from an inertial measurement unit (IMU) sensor;
initiating visual tracking using a camera on said portable device;
capturing a frame using said camera;
determining a plurality of visual features in said frame;
matching said frame to a keyframe, wherein capture of said keyframe precedes capture of said frame, and wherein said keyframe was captured using visual tracking;
computing a rotation amount between said frame and said keyframe;
responsive to a determination that a rotational distance between said frame and said keyframe exceeds a predetermined threshold, promoting said frame to a keyframe status and adding said frame to a first orientation map; and
performing an adjustment of said frame with all prior captured keyframes.

### Claim 13 (depends on 12)

The computer-readable medium as described in claim 12, wherein said determining comprises:
computing repeatable and distinct features in said frame using a feature detecting procedure; and
detecting and describing said features using a feature description procedure.

### Claim 14 (depends on 13)

The computer-readable medium as described in claim 13, wherein said feature detecting procedure is selected from a group consisting of: FAST, Harris & Stephens, Plessey, Shi-Tomasi, Moravec corner detection, Wang and Brady corner detection, and SUSAN corner detector.

### Claim 15 (depends on 13)

The computer-readable medium as described in claim 13, wherein said feature description procedure is selected from a group consisting of: SURF, SIFT and BRIEF.

### Claim 16 (depends on 12)

The computer-readable medium as described in claim 12, further comprising:
responsive to a determination that a rotational distance between said frame and said keyframe is within said predetermined threshold, continuing to search for next keyframe.

### Claim 17 (depends on 12)

The computer-readable medium as described in claim 12, wherein said matching is performed using RANSAC.

### Claim 18 (depends on 12)

The computer-readable medium as described in claim 12, wherein said computing is performed using Horn's procedure.

### Claim 19 (depends on 12)

The computer-readable medium as described in claim 12, wherein said method further comprises:
responsive to a loss of said visual tracking, determining device orientation by combining delta values from said IMU sensor with a last known orientation measurement from said visual tracking; and
building a second orientation map, wherein said second orientation map is created using data from said IMU sensor.

### Claim 20 (depends on 19)

The computer-readable medium as described in claim 19, wherein said method further comprises:
determining if said second orientation map overlaps with keyframes from said first orientation map; and
responsive to a determination of overlap, deleting said second orientation map and continuing to build said first orientation map.

### Claim 21 (depends on 19)

The computer-readable medium as described in claim 19, further comprising:
determining if said second orientation map overlaps with keyframes from said first orientation map; and
responsive to a determination of overlap, merging said second orientation map with said first orientation map.

### Claim 22 (depends on 19)

The computer-readable medium as described in claim 19, further comprising:
determining if said second orientation map overlaps with keyframes from said first orientation map; and
responsive to a determination of no overlap, continuing to build said second orientation map.

### Claim 23

A system for tracking device orientation on a portable device, said system comprising:
a display screen;
a memory;
a camera; and
a processor configured to implement a visual gyroscope, wherein said visual gyroscope performs a method for tracking device orientation on said portable device, wherein said method comprises:
initializing a device orientation to a sensor orientation, wherein said sensor orientation is based on information from an inertial measurement unit (IMU) sensor;
initiating visual tracking using a camera on said portable device;
capturing a frame using said camera;
determining a plurality of visual features in said frame;
matching said frame to a keyframe, wherein capture of said keyframe precedes capture of said frame, and wherein said keyframe was captured using visual tracking;
computing a rotation amount between said frame and said keyframe;
responsive to a determination that a rotational distance between said frame and said keyframe exceeds a predetermined threshold, promoting said frame to a keyframe status and adding said frame to a first orientation map; and
performing an adjustment of said frame with all prior captured keyframes.
