# US 10,962,366 B2 · Visual odometry and pairwise alignment for high definition map creation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,962,366 B2
- Document ID: US10962366B2
- Status: Granted
- Kind: B2
- Application number: 16/787,469
- Filed: 2020-02-11
- Published: 2020-09-10
- Granted: 2021-03-30
- Earliest claimed: 2016-12-30
- Assignees: Deepmap Inc.
- Applicants: DeepMap Inc.
- Inventors: Ronghua Zhang; Chen Chen; Di Zeng; Mark Damon Wheeler
- Web page: https://patentgrep.com/patent/US10962366B2

## CPC

- G06T 17/05: Geographic models
- B60W 40/06: Road conditions
- G01C 11/06: by comparison of two or more pictures of the same area
- G01C 11/12: the pictures being supported in the same relative position as when they were taken
- G01C 11/30: by triangulation
- G01C 21/005: {with correlation of navigation data from several sources, e.g. map or contour matching (G01C21/30 takes precedence)}
- G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera}
- G01C 21/3635: {Guidance using 3D or perspective road maps}
- G01C 21/3694: {Output thereof on a road map}
- G01C 21/3848: {Data obtained from both position sensors and additional sensors}
- G01C 21/3867: {Geometry of map features, e.g. shape points, polygons or for simplified maps}
- G01S 19/42: Determining position
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G06T 7/11: Region-based segmentation
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/248: {involving reference images or patches}
- G06T 7/55: from multiple images
- G06T 7/593: from stereo images
- G06T 7/68: of symmetry
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06T 7/74: {involving reference images or patches}
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G08G 1/20: {Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles}
- B60W 2552/00: Input parameters relating to infrastructure
- B60W 2552/53: Road markings, e.g. lane marker or crosswalk
- G01S 17/89: for mapping or imaging
- G01S 19/46: the supplementary measurement being of a radio-wave signal type
- G01S 19/47: the supplementary measurement being an inertial measurement, e.g. tightly coupled inertial
- G06T 2200/04: involving 3D image data
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20048: Transform domain processing
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T 2207/30256: Lane; Road marking
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T 2215/12: Shadow map, environment map
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

As an autonomous vehicle moves through a local area, pairwise alignment may be performed to calculate changes in the pose of the vehicle between different points in time. The vehicle comprises an imaging system configured to capture image frames depicting a portion of the surrounding area. Features are identified from the captured image frames, and a 3-D location is determined for each identified feature. The features of different image frames corresponding to different points in time are analyzed to determine a transformation in the pose of the vehicle during the time period between the image frames. The determined poses of the vehicle are used to generate an HD map of the local area.

## Claims

### Claim 1

A method, comprising:
receiving a set of image frames depicting a local area of a vehicle, comprising a first image frame corresponding to a first point in time and a second image frame corresponding to a second point in time;
determining a first set of features of the first image frame and a second set of features of the second image frame, each feature of the first and second set of features associated with a respective location within the local area captured by the respective image frame;
determining a transformation between a first position of the vehicle at the first point in time, and a second position of the vehicle at the second point in time, by:
selecting a first subset of features from the first set of features;
identifying a second subset of features from the second set of features, each corresponding to a respective feature of first subset of features, based upon a level of similarity to the respective feature of the first subset of features; and
determining the transformation based upon the respective locations of corresponding features of the first subset of features and the second subset of features; and
updating a high definition map of the local area based on the transformation, the high definition map for use in driving by one or more autonomous vehicles.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the first image frame includes a first image and a second image; and
determining the first set of features is based on a first set of feature points of the first image that match a second set of feature points of the second image.

### Claim 3 (depends on 2)

The method of claim 2, further comprising determining that the first set of feature points match the second set of feature points based on a distance between a first location of the first set of feature points in the first image and a second location of the second feature points in the second image.

### Claim 4 (depends on 3)

The method of claim 3, further comprising determining the respective location of the first set of features within the local area based on the first location of the first set of feature points in the first image and the second location of the second feature points in the second image.

### Claim 5 (depends on 2)

The method of claim 2, further comprising determining that the first set of feature points match the second set of feature points based on a level of similarity between the first set of feature points and the second set of feature points.

### Claim 6 (depends on 5)

The method of claim 5, further comprising determining the level of similarity between the first set of feature points and the second set of feature points based on respective feature descriptors associated with the first set of feature points and the second set of feature points.

### Claim 7 (depends on 1)

The method of claim 1, further comprising determining the level of similarity based on one or more of: a geometric relationship between the first subset of features and the second subset of features, or respective feature descriptors associated with the first subset of features and the second subset of features.

### Claim 8 (depends on 1)

The method of claim 1, further comprising selecting the transformation for updating the high definition map over one or more other transformations based on a determined accuracy of the selected transformation as compared to a respective accuracy of each of the one or more other transformations.

### Claim 9 (depends on 8)

The method of claim 8, further comprising determining the accuracy of the selected transformation based on a number of matches between the first set of features and the second set of features.

### Claim 10

One or more non-transitory computer-readable storage media having instructions stored thereon that, in response to being executed by one or more processors, cause a system to perform operations comprising:
receiving a set of image frames depicting a local area of a vehicle, comprising a first image frame corresponding to a first point in time and a second image frame corresponding to a second point in time;
determining a first set of features of the first image frame and a second set of features of the second image frame, each feature of the first and second set of features associated with a respective location within the local area captured by the respective image frame;
determining a transformation between a first position of the vehicle at the first point in time, and a second position of the vehicle at the second point in time, by:
selecting a first subset of features from the first set of features;
identifying a second subset of features from the second set of features, each corresponding to a respective feature of first subset of features, based upon a level of similarity to the respective feature of the first subset of features; and
determining the transformation based upon the respective locations of corresponding features of the first subset of features and the second subset of features; and
updating a high definition map of the local area based on the transformation, the high definition map for use in driving by one or more autonomous vehicles.

### Claim 11 (depends on 10)

The one or more non-transitory computer-readable storage media of claim 10, wherein:
the first image frame includes a first image and a second image; and
determining the first set of features is based on a first set of feature points of the first image that match a second set of feature points of the second image.

### Claim 12 (depends on 11)

The one or more non-transitory computer-readable storage media of claim 11, the operations further comprising determining that the first set of feature points match the second set of feature points based on a distance between a first location of the first set of feature points in the first image and a second location of the second feature points in the second image.

### Claim 13 (depends on 12)

The one or more non-transitory computer-readable storage media of claim 12, the operations further comprising determining the respective location of the first set of features within the local area based on the first location of the first set of feature points in the first image and the second location of the second feature points in the second image.

### Claim 14 (depends on 11)

The one or more non-transitory computer-readable storage media of claim 11, the operations further comprising determining that the first set of feature points match the second set of feature points based on a level of similarity between the first set of feature points and the second set of feature points.

### Claim 15 (depends on 14)

The one or more non-transitory computer-readable storage media of claim 14, the operations further comprising determining the level of similarity between the first set of feature points and the second set of feature points based on respective feature descriptors associated with the first set of feature points and the second set of feature points.

### Claim 16 (depends on 10)

The one or more non-transitory computer-readable storage media of claim 10, the operations further comprising determining the level of similarity based on one or more of: a geometric relationship between the first subset of features and the second subset of features, or respective feature descriptors associated with the first subset of features and the second subset of features.

### Claim 17 (depends on 10)

The one or more non-transitory computer-readable storage media of claim 10, the operations further comprising selecting the transformation for updating the high definition map over one or more other transformations based on a determined accuracy of the selected transformation as compared to a respective accuracy of each of the one or more other transformations.

### Claim 18 (depends on 17)

The one or more non-transitory computer-readable storage media of claim 17, wherein determining the accuracy of the selected transformation is based on a number of matches between the first set of features and the second set of features.

### Claim 19

A system comprising:
one or more processors; and
one or more non-transitory computer-readable storage media having instructions stored thereon that, in response to being executed by the one or more processors, cause the system to perform operations comprising:
receiving a set of image frames depicting a local area of a vehicle, comprising a first image frame corresponding to a first point in time and a second image frame corresponding to a second point in time;
determining a first set of features of the first image frame and a second set of features of the second image frame, each feature of the first and second set of features associated with a respective location within the local area captured by the respective image frame;
determining a transformation between a first position of the vehicle at the first point in time, and a second position of the vehicle at the second point in time, by:
selecting a first subset of features from the first set of features;
identifying a second subset of features from the second set of features, each corresponding to a respective feature of first subset of features, based upon a level of similarity to the respective feature of the first subset of features; and
determining the transformation based upon the respective locations of corresponding features of the first subset of features and the second subset of features; and
updating a high definition map of the local area based on the transformation, the high definition map for use in driving by one or more autonomous vehicles.

### Claim 20 (depends on 19)

The system of claim 19, wherein:
the first image frame includes a first image and a second image; and
determining the first set of features is based on a first set of feature points of the first image that match a second set of feature points of the second image.

### Claim 21 (depends on 20)

The system of claim 20, further comprising determining that the first set of feature points match the second set of feature points based on a distance between a first location of the first set of feature points in the first image and a second location of the second feature points in the second image.

### Claim 22 (depends on 21)

The system of claim 21, further comprising determining the respective location of the first set of features within the local area based on the first location of the first set of feature points in the first image and the second location of the second feature points in the second image.

### Claim 23 (depends on 20)

The system of claim 20, further comprising determining that the first set of feature points match the second set of feature points based on a level of similarity between the first set of feature points and the second set of feature points.

### Claim 24 (depends on 19)

The system of claim 19, further comprising determining the level of similarity between the first set of feature points and the second set of feature points based on respective feature descriptors associated with the first set of feature points and the second set of feature points.

### Claim 25 (depends on 19)

The system of claim 19, further comprising determining the level of similarity based on one or more of: a geometric relationship between the first subset of features and the second subset of features, or respective feature descriptors associated with the first subset of features and the second subset of features.

### Claim 26 (depends on 19)

The system of claim 19, the operations further comprising selecting the transformation for updating the high definition map over one or more other transformations based on a determined accuracy of the selected transformation as compared to a respective accuracy of each of the one or more other transformations.

### Claim 27 (depends on 26)

The system of claim 26, wherein determining the accuracy of the selected transformation is based on a number of matches between the first set of features and the second set of features.