# US 2019/0095745 A1 · Systems and methods to improve visual feature detection using motion-related data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0095745 A1
- Document ID: US20190095745A1
- Status: Published application
- Kind: A1
- Application number: 16/102,642
- Filed: 2018-08-13
- Published: 2019-03-28
- Earliest claimed: 2017-09-28
- Assignees: Baidu USA LLC
- Applicants: Baidu USA, LLC
- Inventors: Yingze BAO; Mingyu CHEN
- Web page: https://patentgrep.com/patent/US20190095745A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- G06T 7/73: using feature-based methods
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06T 2207/30244: Camera pose
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Described herein are systems and methods that use motion-related data combined with image data to improve the speed and the accuracy of detecting visual features by predict the locations of features using the motion-related data. In embodiments, given a set of features in a previous image frame and given a next image frame, localization of the same set of features in the next image frame is attempted. In embodiments, motion-related data is used to compute the relative pose transformation between the two image frames, and the image location of the features may then be transformed to obtain their location prediction in the next frame. Such a process greatly reduces the search space of the features in the next image frame, and thereby accelerates and improves feature detection.

## Claims

### Claim 1

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes steps for visual feature detection using motion-related data of a mobile system comprising:
receiving motion-related data of the mobile system from a time period at or after capture of a previous image frame to before or at the capture of a current frame;
using at least some of the motion-related data to determine a rotation for motion of the mobile system during at least a portion of the time period;
for each previous feature from a set of previous features in the previous image frame:
given a previous feature in the previous image frame from a set of previous features in the previous image frame and the rotation, determining the previous feature's predicted location and angle in the current image frame;
detecting one or more similar features in the current image frame using at least the previous feature's predicted location;
performing subpixel refinement for at least some of the detected similar features in the current image;
discarding each detected similar feature, following subpixel refinement, whose corner score is below a corner score threshold to obtain a set of remaining features in the current image frame; and
using at least some of the previous features and at least some of the remaining features to update a location and orientation of the mobile system.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium or media of claim 1 wherein the step of using at least some of the previous features and at least some of the remaining features to update a location and orientation of the mobile system comprises:
performing one or more a simultaneous localization and mapping (SLAM) methods using at least some of the previous features and at least some of the remaining features to update a location and orientation of the mobile system.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium or media of claim 1 wherein the step of detecting one or more similar features in the current image frame using at least the previous feature's predicted location comprises:
given a previous feature from the set of previous features and the previous feature's predicted location:
performing optical flow search by starting from the predicted location in the current image frame to identify a candidate feature that is potentially a match to the previous feature;
responsive to an optical flow score for the candidate feature in the current image frame being higher than a difference threshold score, discarding the candidate feature as not being a match for the previous feature; and
responsive to an optical flow score for the candidate feature in the current image frame not being higher than a difference threshold score, classifying the candidate feature as being a match for the previous feature.

### Claim 4 (depends on 3)

The non-transitory computer-readable medium or media of claim 3 wherein the step of responsive to an optical flow score for the candidate feature in the current image frame being higher than a difference threshold score further comprises:
responsive to determining to continue to search for a candidate feature:
searching in the current image frame within a local image patch of the previous feature's predicted location in the current image frame for a candidate feature;
responsive to an optical flow score for the candidate feature in the current image frame being higher than a difference threshold score, discarding the candidate feature as not being a match for the previous feature; and
responsive to an optical flow score for the candidate feature in the current image frame not being higher than a difference threshold score, classifying the candidate feature as being a match for the previous feature; and
responsive to determining to stop searching for a candidate feature, deeming that there is no match found in the current image for the previous feature.

### Claim 5 (depends on 1)

The non-transitory computer-readable medium or media of claim 1 wherein a feature includes a location in an image frame and a descriptor and wherein the step of detecting one or more similar features in the current image frame comprises:
given a previous feature from the set of previous features and the previous feature's predicted location:
identifying a set of candidate feature points within a local neighborhood of the predicted feature's location that have an interest point score or cornerness score above a threshold;
extracting a feature descriptors for each candidate feature from the set of candidate feature points;
selecting a best match feature from the set of candidate feature points based at least upon a comparison of feature descriptors with the previous feature; and
updating the predicted feature's location with an actual location of the best match feature.

### Claim 6 (depends on 5)

The non-transitory computer-readable medium or media of claim 5 further comprising the step of:
responsive to there being no best match feature, discarding the previous feature.

### Claim 7 (depends on 1)

The non-transitory computer-readable medium or media of claim 1 further comprising the step of:
performing one or more feature identification methods on one or more portions of the current image to identify a set of supplemental features.

### Claim 8 (depends on 7)

The non-transitory computer-readable medium or media of claim 7 wherein the one or more portions of the current image selected for feature identification are those portions of the current image that are not within a threshold distance of one of the remaining features.

### Claim 9 (depends on 1)

The non-transitory computer-readable medium or media of claim 1 further comprising the step of:
supplementing the set of remaining features of the current image frame so that a total number of features from the current image frame is within a number range of the set of previous features.

### Claim 10

A processor-implement method for visual feature detection using motion-related data of a mobile system, the method comprising:
receiving motion-related data of the mobile system from a time period at or after capture of a previous image frame to before or at the capture of a current frame;
using at least some of the motion-related data to determine a rotation for motion of the mobile system during at least a portion of the time period;
for each previous feature from a set of previous features in the previous image frame:
given a previous feature in the previous image frame from a set of previous features in the previous image frame and the rotation, determining the previous feature's predicted location and angle in the current image frame;
detecting one or more similar features in the current image frame using at least the previous feature's predicted location;
performing subpixel refinement for at least some of the detected similar features in the current image;
discarding each detected similar feature, following subpixel refinement, whose corner score is below a corner score threshold to obtain a set of remaining features in the current image frame; and
using at least some of the previous features and at least some of the remaining features to update a location and orientation of the mobile system.

### Claim 11 (depends on 10)

The processor-implemented method of claim 10 wherein the step of using at least some of the previous features and at least some of the remaining features to update a location and orientation of the mobile system comprises:
performing one or more a simultaneous localization and mapping (SLAM) methods using at least some of the previous features and at least some of the remaining features to update a location and orientation of the mobile system.

### Claim 12 (depends on 10)

The processor-implemented method of claim 10 wherein the step of detecting one or more similar features in the current image frame using at least the previous feature's predicted location comprises:
given a previous feature from the set of previous features and the previous feature's predicted location:
performing optical flow search by starting from the predicted location in the current image frame to identify a candidate feature that is potentially a match to the previous feature;
responsive to an optical flow score for the candidate feature in the current image frame being higher than a difference threshold score, discarding the candidate feature as not being a match for the previous feature; and
responsive to an optical flow score for the candidate feature in the current image frame not being higher than a difference threshold score, classifying the candidate feature as being a match for the previous feature.

### Claim 13 (depends on 12)

The processor-implemented method of claim 12 wherein the step of responsive to an optical flow score for the candidate feature in the current image frame being higher than a difference threshold score further comprises:
responsive to determining to continue to search for a candidate feature:
searching in the current image frame within a local image patch of the previous feature's predicted location in the current image frame for a candidate feature;
responsive to an optical flow score for the candidate feature in the current image frame being higher than a difference threshold score, discarding the candidate feature as not being a match for the previous feature; and
responsive to an optical flow score for the candidate feature in the current image frame not being higher than a difference threshold score, classifying the candidate feature as being a match for the previous feature; and
responsive to determining to stop searching for a candidate feature, deeming that there is no match found in the current image for the previous feature.

### Claim 14 (depends on 10)

The processor-implemented method of claim 10 wherein a feature includes a location in an image frame and a descriptor and wherein the step of detecting one or more similar features in the current image frame comprises:
given a previous feature from the set of previous features and the previous feature's predicted location:
identifying a set of candidate feature points within a local neighborhood of the predicted feature's location that have an interest point score or cornerness score above a threshold;
extracting a feature descriptors for each candidate feature from the set of candidate feature points;
selecting a best match feature from the set of candidate feature points based at least upon a comparison of feature descriptors with the previous feature; and
updating the predicted feature's location with an actual location of the best match feature.

### Claim 15 (depends on 14)

The processor-implemented method of claim 14 further comprising the step of:
responsive to there being no best match feature, discarding the previous feature.

### Claim 16

A mobile system for visual feature detection using motion-related data of the mobile system comprising:
a vision system for capturing one or more image frames;
one or more sensors for capturing motion-related data;
one or more processors; and
a non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor of the one or more processors, causes steps to be performed comprising:
receiving motion-related data of the mobile system from a time period at or after capture of a previous image frame to before or at the capture of a current frame;
using at least some of the motion-related data to determine a rotation for motion of the mobile system during at least a portion of the time period;
for each previous feature from a set of previous features in the previous image frame:
given a previous feature in the previous image frame from a set of previous features in the previous image frame and the rotation, determining the previous feature's predicted location and angle in the current image frame;
detecting one or more similar features in the current image frame using at least the previous feature's predicted location;
performing subpixel refinement for at least some of the detected similar features in the current image;
discarding each detected similar feature, following subpixel refinement, whose corner score is below a corner score threshold to obtain a set of remaining features in the current image frame; and
using at least some of the previous features and at least some of the remaining features to update a location and orientation of the mobile system.

### Claim 17 (depends on 16)

The mobile system of claim 16 wherein the step of using at least some of the previous features and at least some of the remaining features to update a location and orientation of the mobile system comprises:
performing one or more a simultaneous localization and mapping (SLAM) methods using at least some of the previous features and at least some of the remaining features to update a location and orientation of the mobile system.

### Claim 18 (depends on 16)

The mobile system of claim 16 wherein the step of detecting one or more similar features in the current image frame using at least the previous feature's predicted location comprises:
given a previous feature from the set of previous features and the previous feature's predicted location:
performing optical flow search by starting from the predicted location in the current image frame to identify a candidate feature that is potentially a match to the previous feature;
responsive to an optical flow score for the candidate feature in the current image frame being higher than a difference threshold score, discarding the candidate feature as not being a match for the previous feature; and
responsive to an optical flow score for the candidate feature in the current image frame not being higher than a difference threshold score, classifying the candidate feature as being a match for the previous feature.

### Claim 19 (depends on 18)

The mobile system of claim 18 wherein the step of responsive to an optical flow score for the candidate feature in the current image frame being higher than a difference threshold score further comprises:
responsive to determining to continue to search for a candidate feature:
searching in the current image frame within a local image patch of the previous feature's predicted location in the current image frame for a candidate feature;
responsive to an optical flow score for the candidate feature in the current image frame being higher than a difference threshold score, discarding the candidate feature as not being a match for the previous feature; and
responsive to an optical flow score for the candidate feature in the current image frame not being higher than a difference threshold score, classifying the candidate feature as being a match for the previous feature; and
responsive to determining to stop searching for a candidate feature, deeming that there is no match found in the current image for the previous feature.

### Claim 20 (depends on 16)

The mobile system of claim 16 wherein a feature includes a location in an image frame and a descriptor and wherein the step of detecting one or more similar features in the current image frame comprises:
given a previous feature from the set of previous features and the previous feature's predicted location:
identifying a set of candidate feature points within a local neighborhood of the predicted feature's location that have an interest point score or cornerness score above a threshold;
extracting a feature descriptors for each candidate feature from the set of candidate feature points;
selecting a best match feature from the set of candidate feature points based at least upon a comparison of feature descriptors with the previous feature; and
updating the predicted feature's location with an actual location of the best match feature.
