US 12,592,156 B2Granted

Method and system for on-board localization

AssigneeCalifornia Institute of Technology

InventorsRoland Brockers, Friedrich Alexander Dietsche, Jeff Hugues Luc Delaune, Pedro Proença, Robert Alexander Hewitt, Georgios Georgakis

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

A method and system provide for on-board localization in a unmanned aerial system (UAS). A map image is generated (using previously acquired images) of an area that the UAS is overflying. The map image is then processed by orthorectifying, referencing the map image in a global reference frame, and generating an abstract map by detecting features and locating the features in the global reference frame. The UAS is then localized by acquiring camera images during flight, selecting a subset of the camera images as localization images, detecting on-board image features (in the localization images), mapping features from the detected on-board image features to the abstract map, deleting outliers to determine an estimated 3D pose, and refining the 3D pose. The localized UAS then used to autonomously navigate the UAS.

Claims29

Dependent claims

A method for on-board localization of an unmanned aerial system (UAS), comprising:

(a) generating a map image of an area that the UAS is overflying, wherein:
(i) the map image is generated on-board the UAS using previously acquired images;
and
(ii) the UAS is navigating in a GPS-denied environment at a distance where overflown terrain cannot be assumed to be flat;
(b) processing the map image by:
(i) orthorectifying the map image;
(ii) referencing the map image in a global reference frame by mapping pixel coordinates in the map image to the global reference frame;
(iii) generating an abstract map, wherein the generating comprises:
(A) detecting features enforcing a local minimum feature density in the map image;
and
(B) the features in the abstract map are located within the global reference frame and assigned a 3D position in the global reference frame;
(c) localizing the UAS by performing on-board the UAS:
(i) acquiring camera images during flight by an on-board camera of the UAS;
(ii) selecting localization images from the camera images via a triage process;
(iii) detecting on-board image features in the localization images;
(iv) performing feature association from the detected on-board image features to the abstract map;
(v) deleting matching outliers to determine an estimate of a 3D pose of the UAS in the global reference frame;
(vi) refining the 3D pose of the UAS as an absolute pose of the UAS in the global reference frame;
and
(d) utilizing the localized UAS to autonomously navigate and control the UAS.

17 more elements

An unmanned aerial system (UAS) comprising:

(a) a UAS vehicle comprising:
(i) one or more rotors for operating the UAS vehicle aerially;
(ii) a processor;
(iii) software executed by the processor for conducting onboard autonomy of the UAS vehicle, wherein the software:
(A) generates a map image of an area that the UAS is overflying, wherein:
(1) the map image is generated on-board the UAS using previously acquired images;
and
(2) the UAS is navigating in a GPS-denied environment at a distance where overflown text cannot be assumed to be flat;
(B) processes the map image by:
(1) orthorectifying the map image;
(2) referencing the map image in a global reference frame by mapping pixel coordinates in the map image to the global reference frame;
(3) generating an abstract map, wherein the generating comprises:
detecting features enforcing a local minimum feature density in the map image;
and
the features in the abstract map are located within the global reference frame and assigned a 3D position in the global reference frame;
(C) localizes the UAS by performing on-board the UAS:
(1) acquiring camera images during flight by an on-board camera of the UAS;
(2) selecting localization images from the camera images via a triage process;
(3) detecting on-board image features in the localization images;
(4) performing feature association from the detected on-board image features to the abstract map;
(5) deleting matching outliers to determine an estimate of a 3D pose of the UAS in the global reference frame;
(6) refining the 3D pose of the UAS as an absolute pose of the UAS in the global reference frame;
and
(D) utilizes the localized UAS to autonomously navigate and control the UAS.

21 more elements

A method for on-board localization of an unmanned aerial system (UAS), comprising:

(a) generating a map image of an area that the UAS is overflying, wherein the map image is generated using previously acquired images;
(b) processing the map image by:
(i) orthorectifying the map image;
(ii) referencing the map image in a global reference frame by mapping pixel coordinates in the map image to the global reference frame;
(iii) generating an abstract map, wherein the generating comprises:
(A) detecting features enforcing a local minimum feature density in the map image;
and
(B) the features in the abstract map are located within the global reference frame and assigned a 3D position in the global reference frame;
(c) localizing the UAS by:
(i) acquiring camera images during flight by an on-board camera of the UAS;
(ii) selecting localization images from the camera images via a triage process;
(iii) detecting on-board image features in the localization images;
(iv) performing feature association from the detected on-board image features to the abstract map;
(v) deleting matching outliers to determine an estimate of a 3D pose of the UAS in the global reference frame;
(vi) refining the 3D pose of the UAS as an absolute pose of the UAS in the global reference frame;
(d) utilizing the localized UAS to autonomously navigate and control the UAS;
and
(e) coping with a changing environment between a time the map image was taken and a time the localization images were acquired on-board by:
(i) using temporal integration to identify areas of change within the map image and invalidate such areas;
and
(ii) processing the map image to:
(A) identify areas of feature distribution that are below a defined distribution threshold reflecting potential obstructions or false feature appearance due to introduced artificial texture;
and
(B) invalidate the identified areas.

21 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/312,444Filed 4 May 2023 · us provisional application
Priority: 4 May 2022
Published: US 2023/0360547 A19 Nov 2023
Granted: 31 Mar 2026
Term ends: Not estimatedThe source records don’t support an estimate.