# US 12,217,450 B2 · Vehicle localization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,450 B2
- Document ID: US12217450B2
- Status: Granted
- Kind: B2
- Application number: 17/666,609
- Filed: 2022-02-08
- Published: 2023-08-10
- Granted: 2025-02-04
- Earliest claimed: 2022-02-08
- Assignees: Ford Global Technologies, LLC
- Applicants: Ford Global Technologies, LLC
- Inventors: Ming Xu; Sourav Garg; Michael Milford; Punarjay Chakravarty; Shubham Shrivastava
- Web page: https://patentgrep.com/patent/US12217450B2

## CPC

- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/46: Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56)
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An approximate camera location on a route can be determined by inputting a first image acquired by a vehicle camera to a first convolutional neural network. First image feature points can be extracted from the first image using a feature extraction algorithm. Pose estimation parameters for a pose estimation algorithm can be selected based on the approximate camera location. A six degree-of-freedom (DoF) camera pose can be determined by inputting the first image feature points and second feature points included in a structure-from-motion (SfM) map based on the route to the pose estimation algorithm which is controlled by the pose estimation parameters. A six DoF vehicle pose can be determined based on the six DoF camera pose.

## Claims

### Claim 1

A computer, comprising:
a processor; and
a memory, the memory including instructions executable by the processor to:
determine an approximate camera location on a route by inputting a first image acquired by a camera included in a vehicle to a convolutional neural network;
extract first image feature points from the first image;
select pose estimation parameters for a pose estimation algorithm based on the approximate camera location;
determine a six degree-of-freedom (DoF) camera pose by inputting the first image feature points and second feature points included in a structure-from-motion (SfM) map based on the route to the pose estimation algorithm which is controlled by the pose estimation parameters, wherein the SfM map includes a collection of three-dimensional points visible from the route generated by determining three-dimensional locations of image feature points in global coordinates from the dataset of reference images and combining them using a 3D mapping software program; and
determine a six DoF vehicle pose based on the six DoF camera pose.

### Claim 2 (depends on 1)

The computer of claim 1, the instructions including further instructions to operate a vehicle by determining a vehicle path based on the six DoF vehicle pose.

### Claim 3 (depends on 1)

The computer of claim 1, wherein a dataset of reference images is acquired by a camera included in a vehicle as it travels the route.

### Claim 4 (depends on 3)

The computer of claim 3, wherein the convolutional neural network is trained based on a generic dataset of reference images.

### Claim 5 (depends on 1)

The computer of claim 1, wherein the pose estimation algorithm includes a Perspective-n-Point (P-n-P) algorithm in a Random Sample Consensus (RANSAC) loop.

### Claim 6 (depends on 1)

The computer of claim 1, wherein the pose estimation parameters include one or more of camera selection, image region selection, a local feature matcher, a reprojection error, and a RANSAC scoring function.

### Claim 7 (depends on 1)

The computer of claim 1, wherein the pose estimation parameters are selected based on the approximate camera location with respect to overlapping portions of the route having approximately equal length.

### Claim 8 (depends on 1)

The computer of claim 1, wherein the instructions include further instructions to extract the first image feature points from the first image using a feature extraction algorithm which includes one or more of Superpoint, Reliable and Repeatable Detector and Descriptor, and Scale Invariant Feature Transform.

### Claim 9 (depends on 1)

The computer of claim 1, wherein the six DoF vehicle pose is determined based on the six DoF camera pose by determining a six DoF offset between the vehicle and the camera.

### Claim 10 (depends on 1)

The computer of claim 1, wherein the six DoF camera pose and the six DoF vehicle pose is determined in global coordinates with respect to three orthogonal position axes and three rotations about the three orthogonal position axes, respectively.

### Claim 11

A method, comprising:
determining an approximate camera location on a route by inputting a first image acquired by a camera included in a vehicle to a convolutional neural network;
extracting first image feature points from the first image;
selecting pose estimation parameters for a pose estimation algorithm based on the approximate camera location;
determining a six degree-of-freedom (DoF) camera pose by inputting the first image feature points and second feature points included in a structure-from-motion (SfM) map based on the route to the pose estimation algorithm which is controlled by the pose estimation parameters, wherein the SfM map includes a collection of three-dimensional points visible from the route generated by determining three-dimensional locations of image feature points in global coordinates from the dataset of reference images and combining them using a 3D mapping software program; and
determining a six DoF vehicle pose based on the six DoF camera pose.

### Claim 12 (depends on 11)

The method of claim 11, further comprising operating a vehicle by determining a vehicle path based on the six DoF vehicle pose.

### Claim 13 (depends on 11)

The method of claim 11, wherein a dataset of reference images is acquired by a camera included in a vehicle as it travels the route.

### Claim 14 (depends on 13)

The method of claim 13, wherein the convolutional neural network is trained based on a generic dataset of reference images.

### Claim 15 (depends on 11)

The method of claim 11, wherein the pose estimation algorithm includes a Perspective-n-Point (P-n-P) algorithm in a Random Sample Consensus (RANSAC) loop.

### Claim 16 (depends on 11)

The method of claim 11, wherein the pose estimation parameters include one or more of camera selection, image region selection, a feature matcher, a reprojection error, and a RANSAC scoring function.

### Claim 17 (depends on 11)

The method of claim 11, wherein the pose estimation parameters are selected based on the approximate camera location with respect to overlapping portions of the route having approximately equal length.

### Claim 18 (depends on 11)

The method of claim 11, wherein extracting the first image feature points from the first image includes using a feature extraction algorithm which includes one or more of Superpoint, Reliable and Repeatable Detector and Descriptor, and Scale Invariant Feature Transform.

### Claim 19 (depends on 11)

The method of claim 11, wherein the six DoF vehicle pose is determined based on the six DoF camera pose by determining a six DoF offset between the vehicle and the camera.

### Claim 20 (depends on 11)

The method of claim 11, wherein the six DoF camera pose and the six DoF vehicle pose is determined in global coordinates with respect to three orthogonal position axes and three rotations about the three orthogonal position axes, respectively.