# US 2025/0166346 A1 · Autonomous vision-based georegistration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166346 A1
- Document ID: US20250166346A1
- Status: Published application
- Kind: A1
- Application number: 18/766,823
- Filed: 2024-07-09
- Published: 2025-05-22
- Earliest claimed: 2023-07-17
- Assignees: Technology Innovation Institute - Sole Proprietorship LLC
- Applicants: Technology Innovation Institute-Sole Proprietorship LLC
- Inventors: Daniel Tortei; Eloy Roura; Najwa Aaraj
- Web page: https://patentgrep.com/patent/US20250166346A1

## CPC

- G06N 3/045: Combinations of networks
- G06V 10/74: Image or video pattern matching; Proximity measures in feature spaces
- G01C 21/005: {with correlation of navigation data from several sources, e.g. map or contour matching (G01C21/30 takes precedence)}
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/096: Transfer learning
- G06T 7/74: {involving reference images or patches}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/757: {Matching configurations of points or features}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 20/13: Satellite images
- G06V 20/17: taken from planes or by drones
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06T 2207/10032: Satellite or aerial image; Remote sensing
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems, methods, and computer-readable media for aerial localization in the absence of satellite (e.g., global navigation satellite system (GNSS)) signals while flying over heterogeneous environments. The present embodiments provide a training strategy for robust cross-seasonal image matching in complex, real-world environments. The challenge put forward is to successfully localize a UAV flying over both densely and sparsely populated areas by means of image registration. To this end, the focus of the present embodiments can be on devising loss components to a visual recognition network that adequately embeds the diverse feature types on the ground. A normalized error metric can be provided as well in order to render a conceptually different method comparable to the present approach. The present technical contributions can provide robust visual recognition algorithms in challenging natural environments.

## Claims

### Claim 1

A method for training an aerial image matching model and georeferencing image data using a trained aerial image matching model, the method comprising:
obtaining a set of training data that includes at least a set of aerial images and a set of satellite reference images, wherein each of the set of aerial images and the set of satellite reference images are georeferenced to uniquely identify position information depicted in each image, wherein each of the set of satellite reference images comprise map-view images forming an aerial map of an area, and wherein each of the set of aerial images comprise frames from an imaging device mounted on an aerial vehicle, and wherein the set of aerial images are stitched to obtain a same aerial map formed in the set of satellite reference images;
training an aerial image matching model by:
sampling one or more triplets, each triplet comprising a set of three image patches of a same dimension, wherein the set of three image patches are randomly cropped from any of the set of aerial images and the set of satellite images;
randomly sampling a center location of each anchor of the triplets, wherein at least a length of a radius distance is maintained between a pair of anchor patches from different triplets;
sampling positive patches of each of the one or more triplets by randomly sampling a location within a predefined radius of the center location;
sampling negative patches of each of the one or more triplets by sampling a location exceeding the predefined radius of the center location; and
computing a network loss by:
generating a soft margin loss as a logarithmic function of an amount of anchor-sample overlap and a difference between a Euclidean distance of embeddings of anchor-positive patches and a Euclidean distance of embeddings of anchor-negative patches within the triplets, wherein the anchor-sample overlap and the difference between the Euclidean distances are weighted by a convergence factor; and
generating a regularization loss as a logarithmic function of a kernel and an embedding, wherein the network loss is a weighted sum of the soft margin loss and the regularization loss weighted by a weighting factor;
further training the aerial image matching model using at least the computed network loss; and
identifying, using the trained aerial image matching model, a best match of any of the set of aerial images based on a comparison between any of the set of aerial images and the set of satellite reference images.

### Claim 2 (depends on 1)

The method of claim 1, wherein the training of the aerial image matching model further comprises:
augmenting any of the sampled triplets by performing any of: a perspective transform, a pure rotation, a scaling, and/or a blurring process.

### Claim 3 (depends on 1)

The method of claim 1, wherein each anchor point specifies a georeferenced position in an environment, and wherein a subset of the set of satellite reference images have georeferenced positions within the radius of each anchor point.

### Claim 4 (depends on 1)

The method of claim 1, wherein the aerial image matching model is a convolutional neural network (CNN) model.

### Claim 5 (depends on 4)

The method of claim 4, wherein the CNN model includes one aggregation layer of a vector of locally aggregated descriptors (netVLAD) with at least 32 clusters.

### Claim 6 (depends on 4)

The method of claim 4, wherein the CNN model includes a fully connected layer with an output embedding dimension which include values of 1024, 2048, or 4096.

### Claim 7 (depends on 1)

The method of claim 1, wherein the set of aerial images are obtained from an image device onboard the aerial vehicle, and wherein the set of aerial images are nadir images of an environment from an altitude above a ground level.

### Claim 8 (depends on 1)

The method of claim 1, wherein the set of aerial images are part of a dataset that includes multiple terrains and both urban and rural environments.

### Claim 9 (depends on 1)

The method of claim 1, wherein the set of satellite reference images are captured during multiple seasons.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
determining, for the trained aerial image matching model and for at least one other image matching model, a matched Euclidean error and an upper boundary on a matching error in matching any of the set of aerial images to any of the set of satellite reference images; and
calculating a normalized error in matching any of the set of aerial images to any of the set of satellite reference images for both of the trained aerial image matching model and the at least one other image matching model based on the matched Euclidean error and the upper boundary of the matching error determined for both of the trained aerial image matching model and the at least one other image matching model wherein the normalized error provides an objective comparison of different image matching techniques.

### Claim 11

A system comprising:
an imaging device onboard an aerial vehicle;
a computer in electrical communication with the imaging device, where the computer is operative to:
obtain a set of aerial images from the imaging device;
obtain a set of satellite reference images from a dataset, wherein each of the set of satellite reference images are georeferenced to uniquely identify position information depicted in each of the set of satellite reference images; and
identify, using a trained aerial image matching model, a position of any of the set of aerial images based on a comparison between any of the set of aerial images and the georeferenced set of satellite reference images, wherein training the aerial image matching model comprises:
sampling one or more triplets, each triplet comprising a set of three image patches of a same dimension, wherein the set of three image patches are randomly cropped from any of the set of aerial images and the set of satellite images;
randomly sampling a center location of each anchor of the triplets, wherein at least a length of a radius distance is maintained between a pair of anchor patches from different triplets;
sampling positive patches of each of the one or more triplets by randomly sampling a location within a predefined radius of the center location;
sampling negative patches of each of the one or more triplets by sampling a location exceeding the predefined radius of the center location; and
compute a network loss by:
generating a soft margin loss as a logarithmic function of an amount of anchor-sample overlap and a difference between a Euclidean distance of embeddings of anchor-positive patches and a Euclidean distance of embeddings of anchor-negative patches within the triplets, wherein the anchor-sample overlap and the difference between the Euclidean distances are weighted by a convergence factor; and
generating a regularization loss as a logarithmic function of a kernel and an embedding, wherein the network loss is a weighted sum of the soft margin loss and the regularization loss weighted by a weighting factor;
further train the aerial image matching model using at least the computed network loss; and
identify, using the trained aerial image matching model, a best match of any of the set of aerial images based on a comparison between any of the set of aerial images and the set of satellite reference images.

### Claim 12 (depends on 11)

The system of claim 11, wherein the aerial image matching model is a convolutional neural network (CNN) model, wherein the CNN model includes one aggregation layer including a vector of locally aggregated descriptors (netVLAD) with at least 32 clusters, and wherein the CNN model includes a fully connected layer with an output embedding dimension of 1024, 2048 or 4096.

### Claim 13 (depends on 11)

The system of claim 11, wherein the set of aerial images are nadir images of an environment from an altitude above a ground level.

### Claim 14 (depends on 11)

The system of claim 11, wherein the set of training images are part of a dataset that includes multiple terrains and both urban and rural environments, and wherein the set of satellite reference images are obtained from any of a plurality of satellite image sources, and wherein the set of training images captured during multiple seasons.

### Claim 15 (depends on 11)

The system of claim 11, wherein the computer is further operative to:
determine, for the trained aerial image matching model and for at least one other image matching model, a matched Euclidean error and an upper boundary on a matching error in matching any of the set of aerial images to any of the set of satellite reference images; and
calculate a normalized error in matching any of the set of aerial images to any of the set of satellite reference images for both of the trained aerial image matching model and the at least one other image matching model based on the matched Euclidean error and the upper boundary of the matching error determined for both of the trained aerial image matching model and the at least one other image matching model, wherein the normalized error provides an objective comparison of different image matching techniques.

### Claim 16

A computer-readable storage medium containing program instructions for a method being executed by an application, the application comprising code for one or more components that are called by the application during runtime, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to perform steps comprising:
obtaining a set of training data, the set of training data including at least a set of training images and a set of satellite reference images, wherein each of the set of satellite reference images are georeferenced to uniquely identify position information depicted in each of the set of satellite reference images, and wherein set of training images are part of a dataset that includes multiple terrains and both urban and rural environments with the set of training images captured during multiple seasons;
training an aerial image matching model by:
sampling one or more triplets, each triplet comprising a set of three image patches of a same dimension, wherein the set of three image patches are randomly cropped from any of the set of aerial images and the set of satellite images;
randomly sampling a center location of each anchor of the triplets, wherein at least a length of a radius distance is maintained between a pair of anchor patches from different triplets;
sampling positive patches of each of the one or more triplets by randomly sampling a location within a predefined radius of the center location;
sampling negative patches of each of the one or more triplets by sampling a location exceeding the predefined radius of the center location; and
computing a network loss by:
generating a soft margin loss as a logarithmic function of an amount of anchor-sample overlap and a difference between a Euclidean distance of embeddings of anchor-positive patches and a Euclidean distance of embeddings of anchor-negative patches within the triplets, wherein the anchor-sample overlap and the difference between the Euclidean distances are weighted by a convergence factor; and
generating a regularization loss as a logarithmic function of a kernel and an embedding, wherein the network loss is a weighted sum of the soft margin loss and the regularization loss weighted by a weighting factor;
further training the aerial image matching model using at least the computed network loss; and
identifying, using the trained aerial image matching model, a best match of any of the set of aerial images based on a comparison between any of the set of aerial images and the set of satellite reference images.

### Claim 17 (depends on 16)

The computer-readable storage medium of claim 16, wherein the training of the aerial image matching model further comprises:
augmenting any of the sampled triplets by performing any of: a perspective transform, a pure rotation, a scaling, and/or a blurring process.

### Claim 18 (depends on 16)

The computer-readable storage medium of claim 16, wherein the aerial image matching model is a convolutional neural network (CNN) model, wherein the CNN model includes one aggregation layer include a vector of locally aggregated descriptors (netVLAD) with at least 32 clusters, and wherein the CNN model includes a fully connected layer with an output embedding dimension of either 1024, 2048 or 4096.

### Claim 19 (depends on 16)

The computer-readable storage medium of claim 16, wherein the set of aerial images are obtained from an image device onboard an unmanned aerial vehicle, and wherein the set of aerial images are ground-view images of an environment from an altitude above a ground level.

### Claim 20 (depends on 16)

The computer-readable storage medium of claim 16, wherein the computer is further operative to:
determine, for the trained aerial image matching model and for at least one other image matching model, a matched Euclidean error and an upper boundary on a matching error in matching any of the set of aerial images to any of the set of satellite reference images; and
calculate a normalized error in matching any of the set of aerial images to any of the set of satellite reference images for both of the trained aerial image matching model and the at least one other image matching model based on the matched Euclidean error and the upper boundary of the matching error determined for both of the trained aerial image matching model and the at least one other image matching model.
