# US 2025/0155254 A1 · Localization with point to line matching

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0155254 A1
- Document ID: US20250155254A1
- Status: Published application
- Kind: A1
- Application number: 18/508,649
- Filed: 2023-11-14
- Published: 2025-05-15
- Earliest claimed: 2023-11-14
- Assignees: Ford Global Technologies, LLC; The Australian National University
- Applicants: Ford Global Technologies, LLC; Australian National University
- Inventors: Shan Wang; Yanhao Zhang; Akhil Perincherry; Ankit Girish Vora; Hongdong Li
- Web page: https://patentgrep.com/patent/US20250155254A1

## CPC

- G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera}
- G06T 7/73: using feature-based methods
- G01C 21/005: {with correlation of navigation data from several sources, e.g. map or contour matching (G01C21/30 takes precedence)}
- G01C 21/3407: {specially adapted for specific applications}
- G01C 21/3807: {characterised by the type of data}
- G01C 21/3833: {characterised by the source of data}
- G06T 7/50: Depth or shape recovery
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 2207/10032: Satellite or aerial image; Remote sensing
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A computer that includes a processor and a memory, the memory including instructions executable by the processor to determine top key points from one of an aerial feature map or one or more ground feature maps and project the top key points as corresponding lines on the other of the aerial feature map or the one or more ground feature maps. The memory includes instructions to determine a depth estimate for the top key points on the corresponding lines. A high-definition estimated three degree-of-freedom pose of a ground view camera is determined in global coordinates by iteratively determining geometric correspondence between the top key points and the corresponding lines until a global loss function is less than a user determined threshold.

## Claims

### Claim 1

A system, comprising:
a computer that includes a processor and a memory, the memory including instructions executable by the processor to:
determine top key points from one of an aerial feature map or one or more ground feature maps;
project the top key points as corresponding lines on the other of the aerial feature map or the one or more ground feature maps;
determine a depth estimate for the top key points on the corresponding lines; and
determine a high-definition estimated three degree-of-freedom pose of a ground view camera in global coordinates by iteratively determining geometric correspondence between the top key points and the corresponding lines until a global loss function is less than a user determined threshold.

### Claim 2 (depends on 1)

The system of claim 1, wherein the global loss function is determined by summing 1) a pose aware branch loss function determined by calculating a triplet loss between the top key points and the corresponding lines and 2) a recursive pose refine branch loss function determined by calculating a residual between the top key points and the corresponding lines using a Levenberg-Marquardt algorithm.

### Claim 3 (depends on 2)

The system of claim 2, wherein the pose aware branch loss function determines a feature residual based on the determined high-definition estimated three degree-of-freedom pose of the ground view camera and a ground truth three degree-of-freedom pose.

### Claim 4 (depends on 1)

The system of claim 1, wherein the instructions further comprise instructions to determine the one or more ground feature maps and one or more ground attention maps from one or more ground view images with one or more neural networks and determine the aerial feature map and an aerial attention map from an aerial view image with the one or more neural networks.

### Claim 5 (depends on 4)

The system of claim 4, wherein the instructions further comprise instructions to weight the feature maps with the attention maps.

### Claim 6 (depends on 1)

The system of claim 1, wherein the instructions to determine the top key points include instructions to determine the top key points from the one or more ground feature maps.

### Claim 7 (depends on 1)

The system of claim 1, wherein the instructions to determine the high-definition estimated three degree-of-freedom pose of the ground view camera include instructions to determine the high-definition estimated three degree-of-freedom pose based on an initial estimate of the three degree-of-freedom pose of the ground view camera.

### Claim 8 (depends on 1)

The system of claim 1, wherein the aerial view image is a satellite image.

### Claim 9 (depends on 1)

The system of claim 1, wherein the instructions further comprise instructions to output the high-definition estimated three degree-of-freedom pose of the ground view camera to operate a vehicle.

### Claim 10 (depends on 9)

The system of claim 9, further comprising a vehicle computer configured to determine a vehicle path upon which to operate the vehicle based on the high-definition estimated three degree-of-freedom pose of the ground view camera and the aerial view image.

### Claim 11

A method, comprising:
determining top key points from one of an aerial feature map or one or more ground feature maps;
projecting the top key points as corresponding lines on the other of the aerial feature map or the one or more ground feature maps;
determining a depth estimate for the top key points on the corresponding lines; and
determining a high-definition estimated three degree-of-freedom pose of a ground view camera in global coordinates by iteratively determining geometric correspondence between the top key points and the corresponding lines until a global loss function is less than a user determined threshold.

### Claim 12 (depends on 11)

The method of claim 11, wherein the global loss function is determined by summing 1) a pose aware branch loss function determined by calculating a triplet loss between the top key points and the corresponding lines and 2) a recursive pose refine branch loss function determined by calculating a residual between the top key points and the corresponding lines using a Levenberg-Marquardt algorithm.

### Claim 13 (depends on 12)

The method of claim 12, wherein the pose aware branch loss function determines a feature residual based on the determined high-definition estimated three degree-of-freedom pose of the ground view camera and a ground truth three degree-of-freedom pose.

### Claim 14 (depends on 11)

The method of claim 11, further comprising determining the one or more ground feature maps and one or more ground attention maps from one or more ground view images with one or more neural networks and determining the aerial feature map and an aerial attention map from an aerial view image with the one or more neural networks.

### Claim 15 (depends on 14)

The method of claim 14, further comprising weighting the feature maps with the attention maps.

### Claim 16 (depends on 11)

The method of claim 11, wherein the top key points are determined from the ground feature maps.

### Claim 17 (depends on 11)

The method of claim 11, wherein the determined high-definition estimated three degree-of-freedom pose of the vehicle camera is determined based on an initial estimate of the three degree-of-freedom pose of the ground view camera.

### Claim 18 (depends on 11)

The method of claim 11, wherein the one or more neural networks have a U-Net architecture.

### Claim 19 (depends on 11)

The method of claim 11, further comprising outputting the high-definition estimated three degree-of-freedom pose of the ground view camera to operate a vehicle.

### Claim 20 (depends on 19)

The method of claim 19, further comprising determining a vehicle path upon which to operate the vehicle based on the high-definition estimated three degree-of-freedom pose of the ground view camera and the aerial view image.
