# US 12,602,924 B2 · Method for semantic localization of an unmanned aerial vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,924 B2
- Document ID: US12602924B2
- Status: Granted
- Kind: B2
- Application number: 17/710,516
- Filed: 2022-03-31
- Published: 2023-10-05
- Granted: 2026-04-14
- Earliest claimed: 2022-03-31
- Assignees: Wing Aviation LLC
- Applicants: Wing Aviation LLC
- Inventors: Ali Shoeb; Kyle Kakligian; Damien Jourdan; Michael Zhong; Ke Xu; Jeremie Gabor
- Web page: https://patentgrep.com/patent/US12602924B2

## CPC

- G05D 1/101: {specially adapted for aircraft}
- G06V 20/17: taken from planes or by drones
- G01C 21/005: {with correlation of navigation data from several sources, e.g. map or contour matching (G01C21/30 takes precedence)}
- G01C 21/20: Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence)
- G01C 21/3881: {Tile-based structures}
- G01C 21/3889: {Transmission of selected map data, e.g. depending on route}
- G01S 5/16: using electromagnetic waves other than radio waves
- G01S 19/15: Aircraft landing systems
- G01S 19/485: {whereby the further system is an optical system or imaging system}
- G06N 20/00: Machine learning
- G06T 7/74: {involving reference images or patches}
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- B64U 2201/10: autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR

## Abstract

A computer-implemented method comprises receiving an image captured by a camera on an unmanned aerial vehicle (UAV). The image depicts an environment below the UAV. A feature mask associated with the image is generated via a machine learning model that is trained to identify and semantically label pixels representing the environment depicted in the image. One or more reference tiles associated with the environment are retrieved. The reference tiles are associated with particular geographic locations and specify semantically labeled pixels representing the geographic locations. The semantically labeled pixels of the feature mask are correlated with the semantically labeled pixels of at least one of the one or more reference tiles to determine the geographic location of the UAV in the environment.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving an image captured by a camera on an unmanned aerial vehicle (UAV), wherein the image depicts an environment below the UAV;
generating a feature mask associated with the image via a machine learning model that is trained to identify and semantically label pixels representing the environment depicted in the image;
retrieving one or more reference tiles associated with the environment, wherein the one or more reference tiles are associated with particular geographic locations and specify semantically labeled pixels representing the geographic locations; and
correlating semantic labels of the semantically labeled pixels of the feature mask with semantic labels of the semantically labeled pixels of at least one of the one or more reference tiles to determine a geographic location of the UAV in the environment, wherein the correlating comprises:
generating a likelihood grid that indicates relative likelihoods that the location of the UAV is at particular locations of the one or more reference tiles;
generating, based on a plurality of likelihood grids associated with a corresponding plurality of images successively captured as the UAV moves through the environment, a cumulative likelihood grid that indicates relative likelihoods that the location of the UAV is at particular locations of the one or more reference tiles; and
when the relative likelihood associated with a particular location of the one or more reference tiles exceeds a threshold likelihood in the cumulative likelihood grid, determining the geographic location of the UAV based at least in part on the particular location.

### Claim 2 (depends on 1)

The computer-implemented method according to claim 1, wherein correlating the semantically labeled pixels of the feature mask with the semantically labeled pixels of the at least one of the one or more reference tiles comprises:
resizing the feature mask to have a scale that matches a scale associated with the one or more reference tiles.

### Claim 3 (depends on 1)

The computer-implemented method according to claim 1, wherein correlating the semantically labeled pixels of the feature mask with the semantically labeled pixels of the at least one of the one or more reference tiles comprises:
aligning a heading associated with the feature mask to match a heading associated with the one or more reference tiles.

### Claim 4 (depends on 1)

The computer-implemented method according to claim 1, wherein correlating the semantically labeled pixels of the feature mask with the semantically labeled pixels of the at least one of the one or more reference tiles comprises:
combining a plurality of the one or more reference tiles; and
correlating the semantically labeled pixels of the feature mask with semantically labeled pixels of a portion of the combined reference tiles to determine the location of the UAV in the environment.

### Claim 5 (depends on 1)

The computer-implemented method according to claim 1, further comprising validating the determined location of the UAV based on one or more probability diameters at one or more corresponding probability levels in the cumulative likelihood grid.

### Claim 6 (depends on 1)

The computer-implemented method according to claim 1, further comprising controlling the UAV to navigate in the environment based on the determined location of the UAV in the environment.

### Claim 7 (depends on 1)

The computer-implemented method according to claim 1, further comprising:
controlling the UAV to navigate in the environment using a Global Navigation Satellite System (GNSS) system;
detecting a disruption in service from the GNSS system, wherein the geographic location of the UAV in the environment is determined responsive to detecting the disruption in service from the GNSS system; and
subsequent to detecting the disruption in service from the GNSS system, controlling the UAV to navigate in the environment based on the determined location of the UAV in the environment.

### Claim 8 (depends on 1)

The computer-implemented method according to claim 1, further comprising:
controlling the UAV to navigate in the environment using a GNSS system; and
using the determined location of the UAV in the environment to cross-check location data from the GNSS system.

### Claim 9 (depends on 1)

The computer-implemented method according to claim 1, further comprising:
determining a GNSS location of the UAV in the environment using a GNSS system;
determining a refined location of the UAV in the environment based on the GNSS location of the UAV in the environment and the determined location of the UAV in the environment; and
controlling the UAV to navigate in the environment based on the refined location of the UAV in the environment.

### Claim 10 (depends on 1)

The computer-implemented method according to claim 1, wherein generating the feature mask associated with the image via the machine learning model that is trained to identify and semantically label the pixels representing the environment depicted in the image comprises:
generating a feature mask associated with the image via a machine learning model that is trained to identify and semantically label the pixels representing the environment depicted in the image as corresponding to at least one of: a building, a road, vegetation, a vehicle, a driveway, a lawn, or a sidewalk.

### Claim 11 (depends on 1)

The computer-implemented method according to claim 1, further comprising retrieving the one or more reference tiles in advance of a flight of the UAV, wherein the one or more reference tiles are selected based on a planned flight path of the UAV.

### Claim 12 (depends on 1)

The computer-implemented method according to claim 1, further comprising applying a Kalman filter to the determined location of the UAV in the environment to control navigation of the UAV in the environment.

### Claim 13 (depends on 1)

The computer-implemented method according to claim 1, wherein the machine learning model has been trained based on ground truth semantic data, wherein the ground truth semantic data is based on operator labeling of images captured by one or more UAVs.

### Claim 14

An unmanned aerial vehicle (UAV), comprising:
a camera; and
a control system configured to perform operations comprising:
receiving an image captured by the camera, wherein the image depicts an environment below the UAV;
generating a feature mask associated with the image via a machine learning model that is trained to identify and semantically label pixels representing the environment depicted in the image;
retrieving one or more reference tiles associated with the environment, wherein the one or more reference tiles are associated with particular geographic locations and specify semantically labeled pixels representing the geographic locations; and
correlating semantic labels of the semantically labeled pixels of the feature mask with semantic labels of the semantically labeled pixels of at least one of the one or more reference tiles to determine a geographic location of the UAV in the environment, wherein the correlating comprises:
generating a likelihood grid that indicates relative likelihoods that the location of the UAV is at particular locations of the one or more reference tiles;
generating, based on a plurality of likelihood grids associated with a corresponding plurality of images successively captured as the UAV moves through the environment, a cumulative likelihood grid that indicates relative likelihoods that the location of the UAV is at particular locations of the one or more reference tiles; and
when the relative likelihood associated with a particular location of the one or more reference tiles exceeds a threshold likelihood in the cumulative likelihood grid, determining the geographic location of the UAV based at least in part on the particular location.

### Claim 15 (depends on 14)

The UAV according to claim 14, wherein correlating the semantically labeled pixels of the feature mask with the semantically labeled pixels of the at least one of the one or more reference tiles comprises:
combining a plurality of the one or more reference tiles; and
aligning the semantic labels of the feature mask with a portion of the combined reference tiles to determine the location of the UAV in the environment.

### Claim 16 (depends on 15)

The UAV according to claim 15, wherein correlating the semantically labeled pixels of the feature mask with the semantically labeled pixels of the at least one of the one or more reference tiles comprises:
generating a likelihood grid that indicates relative likelihoods that the location of the UAV is at particular locations of the one or more reference tiles.

### Claim 17

A non-transitory computer readable medium comprising program instructions executable by one or more processors to perform operations, the operations comprising:
receiving an image captured by the camera, wherein the image depicts an environment below the UAV;
generating a feature mask associated with the image via a machine learning model that is trained to identify and semantically label pixels representing the environment depicted in the image;
retrieving one or more reference tiles associated with the environment, wherein the one or more reference tiles are associated with particular geographic locations and specify semantically labeled pixels representing the geographic locations; and
correlating semantic labels of the semantically labeled pixels of the feature mask with semantic labels of the semantically labeled pixels of at least one of the one or more reference tiles to determine a geographic location of the UAV in the environment, wherein the correlating comprises:
generating a likelihood grid that indicates relative likelihoods that the location of the UAV is at particular locations of the one or more reference tiles;
generating, based on a plurality of likelihood grids associated with a corresponding plurality of images successively captured as the UAV moves through the environment, a cumulative likelihood grid that indicates relative likelihoods that the location of the UAV is at particular locations of the one or more reference tiles; and
when the relative likelihood associated with a particular location of the one or more reference tiles exceeds a threshold likelihood in the cumulative likelihood grid, determining the geographic location of the UAV based at least in part on the particular location.