# US 9,437,000 B2 · Odometry feature matching

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,437,000 B2
- Document ID: US9437000B2
- Status: Granted
- Kind: B2
- Application number: 14/185,616
- Filed: 2014-02-20
- Published: 2015-08-20
- Granted: 2016-09-06
- Earliest claimed: 2014-02-20
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Johnny Lee; Joel Hesch
- Web page: https://patentgrep.com/patent/US9437000B2

## CPC

- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/66: of image moments or centre of gravity
- G06T 7/73: using feature-based methods
- H04W 4/023: {using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds}
- H04W 4/027: {using movement velocity, acceleration information}
- H04W 4/029: Location-based management or tracking services
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30241: Trajectory
- G06T 2207/30244: Camera pose
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

Methods and systems for determining features of interest for following within various frames of data received from multiple sensors of a device are disclosed. An example method may include receiving data from a plurality of sensors of a device. The method may also include determining, based on the data, motion data that is indicative of a movement of the device in an environment. The method may also include as the device moves in the environment, receiving image data from a camera of the device. The method may additionally include selecting, based at least in part on the motion data, features in the image data for feature-following. The method may further include estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and feature-following of the selected features in the images.

## Claims

### Claim 1

A method comprising:
receiving, using a processor, data from a plurality of sensors of a device;
determining, based on the data, motion data that is indicative of a movement of the device in an environment;
as the device moves in the environment, receiving image data from a camera of the device;
selecting, based at least in part on the motion data, features in the image data for feature-following; and
estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and by feature-following of the selected features in the image data:
wherein the plurality of sensors includes an inertial measurement unit (IMU),
wherein the motion data includes data received from the IMU that is indicative of a direction of gravity being applied to the device, and
wherein selecting features in the image data for feature-following comprise:
identifying one or more vertical lines in the image data as a line that is parallel to the direction of gravity or one or more horizontal lines in the image data as a line that is perpendicular to the direction of gravity; and
providing the identified one or more vertical lines or the identified one or more horizontal lines as a given feature for feature-following.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the plurality of sensors further includes a depth sensor,
wherein the motion data includes data received from the depth sensor that is indicative of a plurality of depths associated with the movement of the device, and
wherein selecting features in the image data for feature-following further comprises:
determining, based on respective depths of the plurality of depths associated with the movement of the device, a plurality of cells in the image data that are divided by respective depths of the plurality of depths; and
based on a given cell of the plurality of cells including a number of features less than a threshold number of features, selecting features for feature-following that correspond to the given cell.

### Claim 3 (depends on 1)

The method of claim 1,
wherein selecting features in the image data for feature tracking further comprises:
determining, based on the image data, a plurality of sections of the image data; and
based on a given section of the plurality of sections including a number of features less than a threshold number of features, selecting features for feature-following that correspond to the given section.

### Claim 4 (depends on 3)

The method of claim 3, wherein respective sections of the plurality of sections are defined based on quadrants associated with the image data.

### Claim 5 (depends on 3)

The method of claim 3,
wherein the camera comprises a wide-angle view camera, and
wherein respective sections of the plurality of sections are defined based on angles associated with the image data.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
receiving, using the processor, environmental data from the plurality of sensors of the device, wherein the environmental data includes shape data that defines one or more shapes of objects in the environment; and
selecting, based at least in part on the shape data, the features in the image data for feature-following.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
determining, based on the motion data, an uncertainty of movement of the device that defines a level of uncertainty of a portion of the movement of the device; and
selecting, based on the uncertainly of movement of the device, the features in the image data for feature-following.

### Claim 8 (depends on 7)

The method of claim 7,
wherein the uncertainty of movement of the device comprises an uncertainty of a roll or a pitch of the device, and
wherein selecting the features in the image data for feature-following comprises selecting the identified one or more horizontal lines for feature-following.

### Claim 9 (depends on 7)

The method of claim 7,
wherein the uncertainty of movement of the device comprises an uncertainty of a yaw of the device, and
wherein selecting the features in the image data for feature-following comprises selecting the identified one or more vertical lines for feature-following.

### Claim 10

A non-transitory computer readable memory configured to store instructions that, when executed by a device, cause the device to perform functions comprising:
receiving, at the device, data from a plurality of sensors of the device;
determining, based on the data, motion data that is indicative of a movement of the device in an environment;
as the device moves in the environment, receiving image data from a camera of the device;
selecting, based at least in part on the motion data, features in the image data for feature-following;
estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and by feature-following of the select features in the image data; and
wherein the plurality of sensors includes an inertial measurement unit (IMU),
wherein the motion data includes data received from the IMU that is indicative of a direction of gravity being applied to the device, and
wherein selecting features in the image data for feature-following comprise:
identifying one or more vertical lines in the image data as a line that is parallel to the direction of gravity or one or more horizontal lines in the image data as a line that is perpendicular to the direction of gravity; and
providing the identified one or more vertical lines or the identified one or more horizontal lines as a given feature for feature-following.

### Claim 11 (depends on 10)

The non-transitory computer readable memory of claim 10,
wherein the plurality of sensors includes a depth sensor,
wherein the motion data further includes data received from the depth sensor that is indicative of a plurality of depths associated with the movement of the device, and
wherein selecting features in the image data for feature-following further comprises:
determining, based on respective depths of the plurality of depths associated with the movement of the device, a plurality of cells in the image data that are divided by respective depths of the plurality of depths; and
based on a given cell of the plurality of cells including a number of features less than a threshold number of features, selecting features for feature-following that correspond to the given cell.

### Claim 12 (depends on 10)

The non-transitory computer readable memory of claim 10,
wherein selecting features in the image data for feature-following further comprises:
determining, based on the image data, a plurality of sections of the image data; and
based on a given section of the plurality of sections including a number of features less than a threshold number of features, selecting features for feature-following that correspond to the given section.

### Claim 13 (depends on 10)

The non-transitory computer readable memory of claim 10, further comprising:
receiving, using the device, environmental data from the plurality of sensors of the device, wherein the environmental data includes shape data that defines one or more shapes of objects in the environment; and
selecting, based at least in part on the shape data, the features in the image data for feature-following.

### Claim 14 (depends on 10)

The non-transitory computer readable memory of claim 10, further comprising:
determining, based on the motion data, an uncertainty of movement of the device that defines a level of uncertainty of a portion of the movement of the device; and
selecting, based on the uncertainly of movement of the device, the features in the image data for feature-following.

### Claim 15

A device comprising:
one or more processors; and
data storage configured to store instructions that, when executed by the one or more processors, cause the device to perform functions comprising:
receiving data from a plurality of sensors of the device;
determining, based on the data, motion data that is indicative of a movement of the device in an environment;
as the device moves in the environment, receiving image data from a camera of the device;
selecting, based at least in part on the motion data, features in the image data for feature-following; and
estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and by feature-following of the selected features in the image data;
wherein the plurality of sensors includes an inertial measurement unit (IMU),
wherein the motion data includes data received from the IMU that is indicative of a direction of gravity being applied to the device, and
wherein the function of selecting features in the image data for feature-following comprise:
identifying one or more vertical lines in the image data as a line that is parallel to the direction of gravity or one or more horizontal lines in the image data as a line that is perpendicular to the direction of gravity; and
providing the identified one or more vertical lines or the identified one or more horizontal lines as a given feature for feature-following.

### Claim 16 (depends on 15)

The device of claim 15, wherein the functions further comprise:
receiving, using the one or more processors, environmental data from the plurality of sensors of the device, wherein the environmental data includes shape data that defines one or more shapes of objects in the environment; and
selecting, based at least in part on the shape data, the features in the image data for feature-following.

### Claim 17 (depends on 15)

The device of claim 15, wherein the functions further comprise:
determining, based on the motion data, an uncertainty of movement of the device that defines a level of uncertainty of a portion of the movement of the device; and
selecting, based on the uncertainly of movement of the device, the features in the image data for feature-following.