# US 2024/0257387 A1 · System and method for visual localization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0257387 A1
- Document ID: US20240257387A1
- Status: Published application
- Kind: A1
- Application number: 18/028,544
- Filed: 2021-09-29
- Published: 2024-08-01
- Earliest claimed: 2020-10-01
- Assignees: Rafael Advanced Defense Systems Ltd.
- Applicants: Rafael Advanced Defense Systems Ltd.
- Inventors: Natan PETERFREUND
- Web page: https://patentgrep.com/patent/US20240257387A1

## CPC

- G06N 3/08: Learning methods
- G06T 7/74: {involving reference images or patches}
- G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06V 10/40: Extraction of image or video features
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/766: using regression, e.g. by projecting features on hyperplanes
- G06V 10/82: using neural networks
- G06V 20/13: Satellite images
- G06V 20/17: taken from planes or by drones
- G06T 2207/10032: Satellite or aerial image; Remote sensing
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20084: Artificial neural networks [ANN]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A computer-implemented method for processing a geo-location of an aerial platform, comprises: (a) providing a set of classification stages, which is trained end-to-end to provide a set of cascade decisions, wherein each cascade decision is optimized for a different classification modality and for different spatial resolution; (b) receiving a geo-location query comprising an aerial sensor image, a reference bird's eye view image-map with coordinate system and sensor pose angles measures with respect to the coordinate system; (c) projecting the aerial sensor image onto the coordinate system of the reference image-map using the pose angles measures; (d) matching between the projected sensor image and corresponding image patches in the reference satellite image-map, by applying said set of classification stages in accordance with features extraction from said images and similarity criteria; (e) generating a cascade sequencing, wherein a classification score is aggregated along a cascade path among said set of cascade stages; (f) generating a cascade decision at each stage along the cascade path based on the aggregated classification score and on defined decision logic between successive stages; (g) retrieving a geo-location associated with said aggregated classification decision; and (h) sending the retrieved geo-location to a geo-location requester.

## Claims

### Claim 1

A computer-implemented method for processing a geo-location of an aerial platform, comprising:
providing a set of classification stages, which is trained end-to-end to provide a set of cascade decisions, wherein each cascade decision is optimized for a different classification modality and for different spatial resolution;
receiving a geo-location query comprising an aerial sensor image, a reference bird's eye view image-map with coordinate system and sensor pose angles measures with respect to the coordinate system;
projecting the aerial sensor image onto the coordinate system of the reference image-map using the pose angles measures;
matching between the projected sensor image and corresponding image patches in the reference satellite image-map, by applying said set of classification stages in accordance with features extraction from said images and similarity criteria;
generating a cascade sequencing, wherein a classification score is aggregated along a cascade path among said set of cascade stages;
generating a cascade decision at each stage along the cascade path based on the aggregated classification score and on defined decision logic between successive stages;
retrieving a geo-location associated with said aggregated classification decision; and
sending the retrieved geo-location to a geo-location requester.

### Claim 2 (depends on 1)

The method according to claim 1, comprising:
matching between the projected sensor image and corresponding image patches in the reference satellite image-map, by applying said set of regression stages in accordance with features extraction from said images and similarity criteria;
defining a set of basic anchor shapes each refers to specific height of the sensor image with respect to reference coordinate system of the reference image-map;
generating classification score, where classification refers to a detection type and to an anchor shape;
generating a cascade regression at each stage along the cascade path, providing an estimation of scaling between shapes of defined anchors and the projection of the sensor-image;
deriving the shape of the projected sensor-image based on classification decision of anchor shape and on regression scaling; and
deriving the sensor height based on the regression scaling.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the cascade stages are based on a Siamese-RPN architecture with two inputs, namely the projected image of the sensor-image and the reference image-map, and for each input on a multi feature pyramid extraction design, which comprises parallel paths of feature extraction stages, each of which is tuned for a different detection modality.

### Claim 4 (depends on 1)

The method according to claim 1, wherein feature extraction for reference image-map is executed in advance, and the entire process during sensor-image capture.

### Claim 5 (depends on 1)

The method of claim 1, wherein modality refers to the amount of extracted features, where small amount refers to semantic, and large to texture.

### Claim 6 (depends on 1)

The method of claim 1, wherein the extracted features are adapted to be used as natural semantic representations of the image.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the classification stages are trained end-to-end according to an Optimal Configuration Cascade (OCC), which applies the cascade detection loss for each stage of the cascade.

### Claim 8 (depends on 7)

The method of claim 7, wherein OCC is applied subject to a constraint on computation complexity.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the cascade decision is based on a successive classification logic.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the successive classification logic comprises defined disagreement of decisions with respect to previous stage or non-target decision at a given stage results with non-target decision.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the cascade path comprises parallel paths and sequential paths.

### Claim 12 (depends on 1)

The method according to claim 1, wherein each stage refers to different expressivity of function approximation and to different spatial resolution.

### Claim 13 (depends on 1)

The method according to claim 1, wherein the set of cascade stages is based on a cascade Siamese localization, which comprises:
a) a feature extraction sub-branch that includes a single Backbone network and multiple feature pyramid generators; and
b) an RPN subnet, comprised of a cascade of RPN detectors, which provides an estimate of detected class (cls) and of the bounding box shape (reg).

### Claim 14 (depends on 13)

The method according to claim 13, wherein a feature pyramid generator is characterized by increased expressivity of function approximation and by increased spatial resolution along the pyramid outputs.

### Claim 15 (depends on 1)

The method of claim 1, wherein the aerial platform is an Unmanned Aerial Vehicle (UAE).

### Claim 16

A system for processing a geo-location of an aerial platform, comprising:
Circuitry adapted to operate a set of classification stages, which is trained end-to-end to provide a set of cascade decisions, wherein each cascade decision is optimized for a different classification modality;
Circuitry adapted to receive a geo-location query comprising an aerial sensor image, a reference bird's eye view image-map with coordinate system and measures of the sensor pose angles;
Circuitry adapted to project the aerial sensor image onto the coordinate system of the reference satellite image-map based on pose angles measures;
Logical means adapted to match between the projected sensor image and corresponding image patches in the reference satellite image-map, by applying said set of classification stages in accordance with features extraction from said images and similarity criteria operating on these features;
Circuitry adapted to generate a cascade sequencing, wherein a classification score is aggregated along a cascade path among said set of cascade stages;
Circuitry adapted to generate a cascade decision at each stage along the cascade path based on the aggregated classification score and on defined detection logic between successive stages;
Circuitry adapted to retrieve a geo-location associated with said aggregated classification decision; and
Communication apparatus adapted to send the retrieved geo-location to a geo-location requester.
