US 10,309,778 B2Granted

Visual odometry and pairwise alignment for determining a position of an autonomous vehicle

AssigneeDeepmap Inc.

InventorsRonghua Zhang, Mark Damon Wheeler

Life of this patent

Estimating the term…

Loading patent…

Abstract

As an autonomous vehicle moves through a local area, pairwise alignment may be performed to calculate changes in the pose of the vehicle between different points in time. The vehicle comprises an imaging system configured to capture image frames depicting a portion of the surrounding area. Features are identified from the captured image frames, and a 3-D location is determined for each identified feature. The features of different image frames corresponding to different points in time are analyzed to determine a transformation in the pose of the vehicle during the time period between the image frames. The determined poses of the vehicle are used to generate an HD map of the local area.

Claims20

Dependent claims

A method, comprising:

receiving, from an imaging system comprising one or more cameras mounted on a vehicle, a plurality of image frames, each image frame associated with a local area surrounding the vehicle at a particular point in time, and comprising at least a first image captured using a first camera of the one or more cameras of the imaging system;
for each image frame of the plurality of image frames:
extracting from the first image of the image frame, a first region, the first region corresponding to a region of the first image depicting a section of ground in front of the vehicle;
determining locations of a set of features corresponding to features on the section of ground within the first region, comprising:
identifying a first set of feature points with the first region on the first image of the image frame;
identifying a second set of feature points within the first region on a second image of the image frame captured using a second camera of the one or more cameras, the first and second images forming a stereo pair;
determining the set of features for the image frame, each corresponding to a feature point of the first set of feature points matched with a feature point of the second set of feature points, based upon one or more epipolar constraints;
and
determining a location of each feature of the set of features based upon the corresponding feature points of the first and second set of feature points;
identifying, from the plurality of image frames, a first image frame corresponding to a first point in time, and a second image frame corresponding to a second point in time;
identifying one or more correspondences between one or more features of the set of features of the first image frame, and one or more features of the set of features of the second image frame;
determining a transformation between a first position of the vehicle at the first point in time and a second position of the vehicle at the second point in time, based upon a transformation between the positions of the first and second subsets of features;
determining a position and orientation of the vehicle based upon the determined transformation;
and
controlling a movement of the vehicle using a control signal generated based upon the determined position and orientation of the vehicle.

12 more elements

A computer program product for generating a high definition map of a local area based upon sensor readings of a vehicle traveling through the local area, the computer program product comprising a tangible computer-readable storage medium containing computer program code that when executed causes one or more processors to:

receiving, from an imaging system comprising one or more cameras mounted on a vehicle, a plurality of image frames, each image frame associated with a local area surrounding the vehicle at a particular point in time, and comprising at least a first image captured using a first camera of the one or more cameras of the imaging system;
for each image frame of the plurality of image frames:
extracting from the first image of the image frame, a first region, the first region corresponding to a region of the first image depicting a section of ground in front of the vehicle;
determining locations of a set of features corresponding to features on the section of ground within the first region, comprising:
identifying a first set of feature points with the first region on the first image of the image frame;
identifying a second set of feature points within the first region on a second image of the image frame captured using a second camera of the one or more cameras, the first and second images forming a stereo pair;
determining the set of features for the image frame, each corresponding to a feature point of the first set of feature points matched with a feature point of the second set of feature points, based upon one or more epipolar constraints;
and
determining a location of each feature of the set of features based upon the corresponding feature points of the first and second set of feature points;
identifying, from the plurality of image frames, a first image frame corresponding to a first point in time, and a second image frame corresponding to a second point in time;
identifying one or more correspondences between one or more features of the set of features of the first image frame, and one or more features of the set of features of the second image frame;
determining a transformation between a first position of the vehicle at the first point in time and a second position of the vehicle at the second point in time, based upon a transformation between the positions of the first and second subsets of features;
and
determining a position and orientation of the vehicle based upon the determined transformation;
and
controlling a movement of the vehicle using a control signal generated based upon the determined position and orientation of the vehicle.

13 more elements

A computer system for generating a high definition map of a local area based upon sensor readings of a vehicle traveling through the local area, the computer system comprising:

one or more computer processors;
and
a tangible computer-readable storage medium containing computer program code that when executed causes one or more processors to performs the steps of:
receiving, from an imaging system comprising one or more cameras mounted on a vehicle, a plurality of image frames, each image frame associated with a local area surrounding the vehicle at a particular point in time, and comprising at least a first image captured using a first camera of the one or more cameras of the imaging system;
for each image frame of the plurality of image frames:
extracting from the first image of the image frame, a first region, the first region corresponding to a region of the first image depicting a section of ground in front of the vehicle;
determining locations of a set of features corresponding to features on the section of ground within the first region, comprising:
identifying a first set of feature points with the first region on the first image of the image frame;
identifying a second set of feature points within the first region on a second image of the image frame captured using a second camera of the one or more cameras, the first and second images forming a stereo pair;
determining the set of features for the image frame, each corresponding to a feature point of the first set of feature points matched with a feature point of the second set of feature points, based upon one or more epipolar constraints;
and
determining a location of each feature of the set of features based upon the corresponding feature points of the first and second set of feature points;
identifying, from the plurality of image frames, a first image frame corresponding to a first point in time, and a second image frame corresponding to a second point in time;
identifying one or more correspondences between one or more features of the set of features of the first image frame, and one or more features of the set of features of the second image frame;
determining a transformation between a first position of the vehicle at the first point in time and a second position of the vehicle at the second point in time, based upon a transformation between the positions of the first and second subsets of features;
and
determining a position and orientation of the vehicle based upon the determined transformation;
and
controlling a movement of the vehicle using a control signal generated based upon the determined position and orientation of the vehicle.

16 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/855,144Filed 27 Dec 2017 · us provisional application
Priority: 30 Dec 2016
Published: US 2018/0188027 A15 Jul 2018
Granted: 4 Jun 2019
Term ends: Not estimatedThe source records don’t support an estimate.

Classification51 codes

G06T 17/05
Three-dimensional [3D] modelling for computer graphics: geographic models
B60W 40/06
Related to ambient conditions: road conditions
G01C 11/06
Interpretation of pictures: by comparison of two or more pictures of the same area
G01C 11/12
By comparison of two or more pictures of the same area: the pictures being supported in the same relative position as when they were taken

Family

Application62/441,080Us provisional application · filed 30 Dec 2016
US 2018/0188027 A1Pre-grant publication · published 5 Jul 2018
US 10,309,778 B2This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key