# US 10,928,207 B2 · Camera based localization for autonomous vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,928,207 B2
- Document ID: US10928207B2
- Status: Granted
- Kind: B2
- Application number: 16/290,658
- Filed: 2019-03-01
- Published: 2019-09-05
- Granted: 2021-02-23
- Earliest claimed: 2018-03-02
- Assignees: Deepmap Inc.
- Applicants: DeepMap Inc.
- Inventors: Ronghua Zhang; Lin Yang
- Web page: https://patentgrep.com/patent/US10928207B2

## CPC

- G01C 21/3848: {Data obtained from both position sensors and additional sensors}
- G01C 21/20: Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence)
- G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera}
- G01C 21/3867: {Geometry of map features, e.g. shape points, polygons or for simplified maps}
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- G06F 16/29: Geographical information databases
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06T 2207/30244: Camera pose
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Camera based localization performed to determine a current pose of an autonomous vehicle without the aid of depth sensors such as LiDAR. The vehicle comprises an imaging system configured to capture image frames depicting portions of the surrounding area. Based on an initial pose of the vehicle, edgels corresponding to three-dimensional locations are loaded and mapped to corresponding edge pixels of the captured image frame. A pose of the vehicle is optimized based upon the determined correspondences by identifying a transformation that minimizes a distance between the edgels and their corresponding edge pixels. The determined transformation can be applied to the initial pose to determine an updated pose of the vehicle.

## Claims

### Claim 1

A method, comprising:
receiving, by an autonomous vehicle, from an imaging system mounted on the vehicle, an image frame, the image frame depicting a portion of a local area surrounding the vehicle;
receiving an initial pose of the autonomous vehicle that is an estimate of an actual pose of the autonomous vehicle;
loading a plurality of edgels from map data of a High Definition (HD) map of the local area based upon the initial pose;
analyzing the image frame to identify a plurality of edge pixels within the image frame;
for each respective edgel of a subset of the plurality of edgels:
projecting the respective edgel onto the image frame, based upon the initial pose; and
determining a correspondence between the projected edgel and an edge pixel of the plurality of edge pixels, based upon a distance between the projected edgel and the edge pixel;
determining, based upon the determined correspondences, a transformation that if applied to the subset of edgels reduces an aggregate distance between the subset of edgels and their corresponding edge pixels;
applying the determined transformation to the initial pose to determine an updated pose of the autonomous vehicle with respect to the actual pose; and
providing signals to the vehicle to navigate the autonomous vehicle based on the updated pose of the autonomous vehicle.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the correspondence between the projected edgel and the edge pixel of the plurality of edge pixels is further based on a level of similarity between a gradient direction of the projected edgel and a gradient direction of the corresponding edge pixel.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of edgels is associated with a map, the method further comprising:
receiving, by the vehicle, the map indicating edgels within a local area.

### Claim 4 (depends on 3)

The method of claim 3, wherein the plurality of edgels correspond to edgels indicated in the map having three-dimensional locations within a threshold radius of the location of the initial pose.

### Claim 5 (depends on 1)

The method of claim 1, further comprising organizing the plurality of edgels into a plurality of clusters, each cluster corresponding to two or more edgels within a threshold distance of each other, based upon the respective three-dimensional locations of the two or more edgels.

### Claim 6 (depends on 5)

The method of claim 5, further comprising determining the subset of edgels by downsampling the edgels of each of the plurality of clusters.

### Claim 7 (depends on 6)

The method of claim 6, wherein downsampling the edgels of a first cluster of the plurality of clusters comprises:
dividing the edgels of the first cluster into a plurality of buckets, based upon respective gradient directions of the edgels; and
for each bucket of the plurality of buckets, retaining a first edgel of the bucket having a location closest to the location of the initial pose and a second edgel of the bucket having a location farthest from the location of the initial pose, and sampling the remaining edgels of the bucket based upon distance to the location of the initial pose.

### Claim 8 (depends on 5)

The method of claim 5, further comprising, for a first cluster of the plurality of clusters:
determining structural information of the first cluster, the structural information indicating at least a distance between a first edgel and a second edgels of the cluster; and
based upon a determination that the first edgel and second edgel correspond to different edges within the image frame, inferring a type of feature associated with the cluster based upon the distance between the first and second edgels.

### Claim 9 (depends on 1)

The method of claim 1, further comprising iteratively:
applying the determined transformation to the subset of edgels;
determining a level of convergence based upon the aggregate distance between the transformed subset of edgels and their corresponding edge pixels;
in response to the level of convergence not satisfying a threshold value:
updating the correspondences by determining a corresponding edge pixel for each of the transformed subset of edgels; and
determining an updated transformation to minimize the aggregate distance between the subset of edgels and their corresponding edge pixels, based upon the updated correspondences.

### Claim 10 (depends on 1)

The method of claim 1, wherein analyzing the image frame to identify a plurality of edge pixels within the image frame comprises:
identifying a portion of the image frame corresponding to ground; and
identifying the plurality of edge pixels within the identified portion.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
using the initial pose and a map of a local area to determine a ground normal vector;
wherein determining the transformation comprises restricting one or more components of the determined transformation based upon the determined ground normal vector.

### Claim 12 (depends on 1)

The method of claim 1, wherein generating the map data comprises:
detecting one or more edges in a first image, each edge having one or more pixels each having a corresponding gradient;
receiving a point corresponding to a three-dimensional location of the region, the point being obtained from a scan of the region;
identifying a corresponding pixel of the first image by projecting the point onto the first image;
in response to a determination that the corresponding pixel is a pixel on an edge of the one or more edges, designating the point as an edgel;
determining a gradient direction corresponding to the edgel, based upon the gradient of the point; and
storing information describing the edgel into the map.

### Claim 13

A non-transitory computer readable storage medium storing computer program instructions, the computer program instructions when executed by a computer processor causes the processor to:
receive from an imaging system mounted on an autonomous vehicle, an image frame, the image frame depicting a portion of a local area surrounding the vehicle;
receive an initial pose of the autonomous vehicle that is an estimate of an actual pose of the autonomous vehicle;
load a plurality of edgels from map data of a High Definition (HD) map of the local area based upon the initial pose;
analyze the image frame to identify a plurality of edge pixels within the image frame;
for each respective edgel of a subset of the plurality of edgels:
project the respective edgel onto the image frame, based upon the initial pose; and
determine a correspondence between the projected edgel and an edge pixel of the plurality of edge pixels, based upon a distance between the projected edgel and the edge pixel;
determine, based upon the determined correspondences, a transformation that if applied to the subset of edgels reduces an aggregate distance between the subset of edgels and their corresponding edge pixels;
apply the determined transformation to the initial pose to determine an updated pose of the autonomous vehicle with respect to the actual pose; and
provide signals to the vehicle to navigate the autonomous vehicle based on the updated pose of the autonomous vehicle.

### Claim 14 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein determining the correspondence between the projected edgel and the edge pixel of the plurality of edge pixels is further based on a level of similarity between a gradient direction of the projected edgel and a gradient direction of the corresponding edge pixel.

### Claim 15 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein the plurality of edgels is associated with a map, and wherein the computer program instructions further cause the processor to receive the map indicating edgels within a local area.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the plurality of edgels correspond to edgels indicated in the map having three-dimensional locations within a threshold radius of the location of the initial pose.

### Claim 17 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein the computer program instructions further cause the processor to:
organize the plurality of edgels into a plurality of clusters, each cluster corresponding to two or more edgels within a threshold distance of each other, based upon the respective three-dimensional locations of the two or more edgels.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the computer program instructions further cause the processor to determine the subset of edgels by downsampling the edgels of each of the plurality of clusters.

### Claim 19 (depends on 18)

The non-transitory computer readable storage medium of claim 18, wherein downsampling the edgels of a first cluster of the plurality of clusters comprises:
dividing the edgels of the first cluster into a plurality of buckets, based upon respective gradient directions of the edgels; and
for each bucket of the plurality of buckets, retaining a first edgel of the bucket having a location closest to the location of the initial pose and a second edgel of the bucket having a location farthest from the location of the initial pose , and sampling the remaining edgels of the bucket based upon distance to the location of the initial pose.

### Claim 20 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the computer program instructions further cause the processor to, for a first cluster of the plurality of clusters:
determine structural information of the first cluster, the structural information indicating at least a distance between a first edgel and a second edgels of the cluster; and
based upon a determination that the first edgel and second edgel correspond to different edges within the image frame, infer a type of feature associated with the cluster based upon the distance between the first and second edgels.

### Claim 21 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein the computer program instructions further cause the processor to, iteratively:
apply the determined transformation to the subset of edgels;
determine a level of convergence based upon the aggregate distance between the transformed subset of edgels and their corresponding edge pixels;
in response to the level of convergence not satisfying a threshold value:
update the correspondences by determining a corresponding edge pixel for each of the transformed subset of edgels; and
determine an updated transformation to minimize the aggregate distance between the subset of edgels and their corresponding edge pixels, based upon the updated correspondences.

### Claim 22 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein the computer program instructions further cause the processor to analyze the image frame to identify a plurality of edge pixels within the image frame by:
identifying a portion of the image frame corresponding to ground; and
identifying the plurality of edge pixels within the identified portion.

### Claim 23 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein the computer program instructions further cause the processor to:
use the initial pose and a map of a local area to determine a ground normal vector;
wherein determining the transformation comprises restricting one or more components of the determined transformation based upon the determined ground normal vector.

### Claim 24 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein generating the map data comprises:
detecting one or more edges in a first image, each edge having one or more pixels each having a corresponding gradient;
receiving a point corresponding to a three-dimensional location of the region, the point being obtained from a scan of the region;
identifying a corresponding pixel of the first image by projecting the point onto the first image;
in response to a determination that the corresponding pixel is a pixel on an edge of the one or more edges, designating the point as an edgel;
determining a gradient direction corresponding to the edgel, based upon the gradient of the point; and
storing information describing the edgel into the map.